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ABSTRACT 


Accurate weather forecasts are vital to air combat operations. 
Quantitative assessments of forecasts and their operational impacts are essential 
to improving weather support for war fighters. We adapted an existing U.S. 
Navy, web-based, near-real time system for collecting and analyzing data on the 
performance and operational impacts of military forecasts. We used the adapted 
system to collect and analyze data on Air Force Weather (AFW) forecasts, and 
the planning and execution of flying operations, at six Air Combat Command and 
Pacific Air Forces bases. We analyzed the data to develop quantitative metrics 
of forecast performance and operational impacts. Our results indicate that 
planning weather forecasts (PWFs) have a higher potential for making positive 
contributions to air operations than do mission execution forecasts (MEFs). This 
is notable because AFW units spend significantly less time developing PWFs 
than MEFs. Surface visibility, cloud ceilings, and cloud layers caused most 
negative mission impacts, indicating these phenomena should be a focus of 
future research and training. We found high levels of mission success even 
when forecasts were inaccurate, perhaps due to aircrew and mission flexibility. 
Our analyses revealed a need for improved education of flying units on the 
nature and availability of AFW products. 
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I. INTRODUCTION 


A. BACKGROUND 

Accurate weather forecasts are vital to planning and conducting effective 
combat operations. Focused quantitative analyses of military weather forecasts 
and their operational impacts are essential to ensure high quality weather 
support for war fighters. In today’s resource limited and changing environment, 
more effective and efficient processes and procedures are essential. Improved 
processes, when employed throughout an organization, can save manpower and 
resources and enhance mission accomplishment. In this study, we have 
analyzed some of the challenges facing Air Force weather (AFW) personnel, 
especially the difficulties associated with obtaining operator feedback on mission 
execution forecasts (MEFs) that is required of CWTs (AFMAN 15-129). To 
address these difficulties, we have developed and tested a system to improve the 
feedback process. The system was designed to improve operator feedback to 
the CWTs without increasing CWT workloads. 

1. War on Terrorism 

The United States military has seen many changes in the way business is 
done since September 11, 2001, when the United States experienced horrific 
attacks by terrorists. In the Air Force Chief of Staff’s remarks at the 2005 Air 
Force Defense Strategy and Transformation Seminar Series in Washington D.C. 
on 9 February 2005, General Moseley indicated there were about 30,000 active. 
Guard, and Reserve airmen deployed (Moseley 2005). In an Air Force Policy 
Letter Digest article in December 2005, Air Force Secretary Michael W. Wynne 
said the Air Force is evolving in response to a new global war on terrorism 
(Wynne 2005). Secretary Wynne indicated in this article that the Air Force 
deployed more than 300 aircraft and 24,000 airmen in support of the war on 
terrorism. 

Deploying so many airmen inevitably results in fewer being available to 
sustain home station operations. In many situations, this causes home stations 
to refocus their operations, which results in many processes and procedures 
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being altered. The new processes and procedures will logically be in the 
forefront of a unit’s focus, while previously developed tasks (e.g., required routine 
training, administrative and planning requirements) may be given a lower priority. 
In the AFW career field, some of the less critical and less easily executed tasks 
include the determination of forecast performance metrics and collection of 
customer feedback. These tasks generally do not have firm timelines, and many 
AFW units lack the expertise to perform them well. However, in a working 
environment in which units are faced with increasing deployments and limited 
resources, these tasks can make AFW units more effective and efficient. Thus 
these tasks become even more essential for achieving overall mission success. 
This highlights the need for highly effective and efficient processes that, once 
implemented, can improve effectiveness while reducing needed resources. 

2. Reengineering Air Force Weather 

a. Initiation and Implementation 

In addition to deployments, AFW personnel have seen many 
internal changes in the way business is done over the last ten years intended to 
increase effectiveness while improving war fighter support. In 1997, the Air 
Force director of weather (AF/A30-W) implemented a reengineering plan to 
realign its force structure at the operational and tactical levels of warfare. The 
changes were necessary because manpower cuts, among other factors, caused 
AFW to fall below the critical personnel levels necessary to sustain itself 
(AFWSPV 2004). 

AFW accomplished both a support and a personnel restructuring. 
Part of the restructure was to stand up operational weather squadrons (OWSs) 
focused at the operational level of warfare. The OWS took over production of 
several tasks from the base weather station, such as terminal aerodrome 
forecasts, weather watches, warnings, and advisories. This freed up the combat 
weather teams (CWTs) at military bases to focus on mission-tailored forecasting 
for their supported units (AFWSPV 2004). AFW leadership envisioned that by 
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focusing the base-level weather team on products, such as MEFs, weather 
teams would be more likely to become fully integrated into flying operations 
(AFAA 2005). 

While some units are still trying to fully implement portions of the 
restructuring, many have been hindered in doing so by additional taskings and 
deployments that have been levied as a result of the war on terrorism. Tasks 
such as daily building patrols, terrorism training, 100% identification checks into 
many base facilities that had no checks before, evacuation drills, accountability 
drills, and simple medical training have inundated an already full schedule. This 
has resulted in many personnel and units becoming task saturated. 

As noted in a recent Air Force audit (AFAA 2005), the scramble to 
keep up with demands placed on CWTs by Air Force instructions and directives 
is challenging. Due to these challenges, restructuring has, in some cases, not 
achieved its full potential (AFAA 2005). 

b. Reengineering Chaiienges 

Although the AF/A30-W reengineering plan was implemented with 
extensive guidance, some CWTs are not able to fully implement some aspects of 
the vision. After holding positions out of the AFW career field for six years, I re¬ 
entered it in June 2001 by taking a position as the CWT commander at an Air 
Force installation supporting fighter aircraft. I had previously read the Air Force 
Instructions (AFIs) regarding reengineering and looked forward to stepping into a 
new organizational framework. What soon became apparent was that the 
concept was challenging to implement, both from a CWT and customer 
perspective, due to limited resources, and perhaps resistance to change by some 
personnel. 

It took me nearly two years as a CWT flight commander to integrate 
weather personnel into two of the flying squadrons at my installation. In fact, one 
squadron never did allow this during my time as the CWT flight commander. 
Integration was only possible when squadron and group leadership were willing 
to endure some of the pain involved. Often, when customer leadership changes. 
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so does the integration of the CWT and its weather support. The challenge is to 
institutionalize a process so that when leadership changes, weather support does 
not. Once weather personnel are integrated into the flying squadrons, it is much 
easier to obtain operator feedback. Integration helps customers get to know 
CWT personnel, which in turn helps the CWT get customer feedback. The 
increase in feedback our weather team gained by integrating our personnel into 
the customers’ environment was invaluable. This feedback enabled us to make 
some drastic operator-driven changes to our MEFs, which led to improved 
weather support to the customer. 

Since 2003, the same CWT I led has reverted back to little or no 
integration of weather personnel into the flying squadrons. This is, in part, due to 
decreased manning because of deployments supporting the war on terrorism. 
Because this integration concept might not be possible at many CWTs, our study 
developed and tested a system for gaining operator feedback for individual 
missions that can be easily adapted to almost any CWT. This system was 
designed to help streamline CWT processes and procedures. 

3. Air Force Manual 15-129, Air and Space Weather Operations - 

Processes and Procedures 

CWT operations are outlined in chapter 4 of Air Force Manual 15-129, Air 
and Space Weather Cperations - Processes and Procedures, 2004 (hereafter 
referred to as AFMAN 15-129). This manual applies to all Air Force personnel 
and organizations conducting weather operations. Because each CWT has its 
own unique mission and responsibilities, the manual gives general guidance on 
what needs to be accomplished. Each CWT must tailor its weather products to 
its customers in such a manner that it provides environmental information ready 
for use by customers in making mission planning and execution decisions. This 
support can be provided in a number of ways, but should follow what AFMAN 15- 
129 calls the MEF process (MEFP). The purpose of the MEFP is to “temporally 
and spatially refine forecast products to provide decision-quality environmental 
information for an operational decision-making process” (AFMAN 15-129). 
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The MEFP is a twelve step process: 

1) Determine the what, when, where, who, and how of the primary and 
secondary missions. 

2) Define weather thresholds. 

3) Define products, services, and data types. 

4) Coordinate operations. 

5) Obtain weather situational awareness. 

6) Conduct mission-scale analysis. 

7) Predict mission execution weather parameters. 

8) Tailor forecast parameters to mission. 

9) Disseminate MEF. 

10) Conduct mission watch. 

11) Update forecast products/information. 

12) Conduct MEF verification. 

These 12 steps fall into four phases: 

1) planning and coordinating (determined by CWT leadership; steps 1- 

4 ), 

2) preparing (steps 5-8) 

3) executing (steps 9-10) 

4) follow-up (steps 11-12) 

The MEFP is a process for continuous assessment and improvement of 
weather support. To achieve this improvement, the full process should be 
routinely used and its effects regularly assessed. This is especially true when 
customer missions change, because then the MEFs, and the steps for evaluating 
the MEFs, must also change. In this study, we have focused on the follow-up 
phase of the MEFP for aviation support. This is sometimes regarded as the final 
phase, but is actually the restart of the first phase, and provides weather 
personnel with a foundation on which to build and remodel the MEF. 
a. Update Forecast Products/Information 
The main goal of step eleven is to improve weather support by 
modifying the MEFs based on customer feedback. Customer feedback may 
change as customer leadership changes. But the resulting changes to the MEFs 
should be relatively small because each CWT should have a weather support 
plan in place that outlines the what, when, and how of the CWT’s products and 
processes. These plans help stabilize the CWT’s products and processes, 
ensure well defined customer support, and make CWT operations more efficient. 
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b. Conduct Mission Verification 

Step twelve of the MEFP directs CWTs to perform seven tasks 
(AFMAN 15-129): 

1) Implement systematic procedures to analyze and measure 
accuracy/relevancy of environmental services provided to 
parent/host unit. 

2) Debrief operator. Face-to-face feedback is preferred; 

employ other feedback methods whenever direct feedback 
cannot be obtained. 

3) Disseminate output from weather debriefs or pilot reports 
(PIREPS) to OWSs and other weather team members. 

4) Perform technical verification (evaluate forecast skills, 
under/over forecast, bias, etc.). 

5) Conduct operations verification based on established 
“Go/No Go” thresholds. 

6) Develop and apply metrics to process improvement. Use 
feedback to develop rules of thumb and lessons learned. 

7) Accomplish, document, train with, and cross feed forecast 
reviews. 

The seven tasks for step 12 are all important, but, in practice, they can be 
very difficult to complete, especially given the resource limitations described 
earlier in this chapter. In this study, we focused on developing and testing a 
system to facilitate the completion of four of these tasks: 1, 2, 5, and 6. Our 
goals were to create a system that simplified the process of: (1) collecting data 
with which to verify forecasts and assess customer performance; (2) collecting 
customer feedback data; (3) analyzing forecast and customer performance; (4) 
relating forecast and customer performance to determine the operational impacts 
of the forecasts; and (5) identifying methods for improving weather support for 
customer operations. These five goals directly address tasks 1,2, 5, and 6. 

Implicit in these tasks and goals is the understanding that good forecast 
performance (e.g., high forecast accuracy) does not necessarily indicate good 
customer performance (e.g., success in completing missions). There are many 
ways in which accurate forecasts can fail to provide optimal support for mission 
planning and execution (e.g., by failing to provide relevant forecasts or to provide 
forecast information in a timely, understandable and readily useable format) 
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The preferred method to obtain customer feedback is face-to-face 
discussion (AFMAN 15-129). However, this is not always possible due to 
resource constraints. Further, if a CWT has educated its customers on weather 
impacts and the customers know what feedback is needed, then a face-to-face 
debrief may not be needed. Whatever the method for collecting customer 
feedback, it is important that the feedback comes from the customer and not from 
CWT personnel inferring what the customers would say if asked. The system we 
developed for this study was designed to collect data directly from aircrews on 
the accuracy of the forecasts they received, the success of their missions, the 
usefulness and specific impacts of the forecasts on the planning and execution of 
their missions, the weather phenomena that were encountered during the 
mission, and the impacts of those phenomena on the execution of the mission.. 

4. Air Force Instruction 15-114, Functionai Resource and Weather 
Technical Performance Evaluation 

Guidance and procedures for measuring and evaluating the operational 
effectiveness and technical performance of atmospheric and space 
environmental support, collectively termed weather support, is defined in Air 
Force Instruction 15-114, Functional Resource and Weather Technical 
Performance Evaluation, 2001 (hereafter referred to as AFI 15-114). This 14- 
page document defines three categories of forecast assessment: 

1) forecast impacts on mission execution determined by operational 
verification (OPVER) 

2) forecast accuracy determined by aerodrome forecast verification 

3) resource protection effectiveness determined by warning/advisory 
verification (WARNVER) 

a. OPVER 

AFI 15-114 calls operational verification of the MEF (MEFVER) the “single 
most important mission-oriented, operational effectiveness assessment 
requirement for CWTs.” The instruction further explains that this type of OPVER 
will be used by the Air Staff to explain AFW capabilities to Department of 
Defense and National senior leaders. According to AFI 15-114, “CWTs develop 
OPVER criteria through close coordination with operators. CWTs then collect 
verification data by debriefing customers and/or analyzing observed versus 
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forecast conditions to determine forecast impact on tactical-level mission 
effectiveness.” However, most CWTs do not have the resources or expertise to 
develop such an OPVER process. Our study is intended to help correct this 
problem by streamlining the OPVER process. 

b. Automating Readiness and Technicai Performance 
Metrics 

Paragraph two of API 15-114 calls for a system that, “automates all 
metrics, from data collection, aggregation, to data quality control. The end-state 
will be an automated web-based system with the capability to provide ad hoc 
analyses and reports (assessments) for all levels of AFW support.” The 
instruction also provides guidance on how to proceed until this automation is 
realized. The system we have developed in this study automates many of the 
steps needed to calculate the required metrics. 

5. Air Force Agency Report 

The Air Force Audit Agency (AFAA) issued a “Weather Operations 
Metrics” report on 27 September 2005 (AFAA 2005), in response to a request 
from AF/A30-W, to determine whether or not the Air Force effectively 
implemented MEFVER. Specifically, the AFAA was asked to determine if 
weather personnel effectively: 

1) accomplished mission execution forecast verification 

2) included mission operator feedback on weather forecasts in 

MEFVERs 

3) identified improvement opportunities (AFAA 2005) 

There were 19 CWTs reviewed in the audit, three of which also 
participated in our research project. The audit was performed during February- 
April 2005 using documents dated 13 March 1998 through 17 March 2005. The 
audit found that AFIs do not provide standardized procedures for obtaining 
mission operator feedback on weather forecasts and do not require operations 
personnel to provide formal feedback on weather forecast accuracy and mission 
impact. The audit found that low customer feedback rates had a negative impact 
on weather teams’ efforts to identify improvement opportunities. Specifically, the 
audit mentioned that low response rates from customers hindered data usage 

and prevented weather teams from deriving any usable information from 
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customer feedback. The audit also indicated that the relevant APIs did not 
provide sufficient guidance or clear procedures for identifying improvement 
opportunities within the MEFVER program (AFAA 2005). 

The audit team made several recommendations to the Deputy Chief of 
Staff, Air and Space Operations (AF/XO). The recommendations that apply to 
this study are to: 

1) establish standardized procedures for obtaining mission operator 
feedback on weather forecasts 

2) direct AF/XOO-W and Air Force Director of Operational Training 
(AF/XOO-T) to coordinate and amend existing processes to require 
operators to provide formal feedback on weather forecast accuracy 
and mission impact. 

The AF/A3 concurred with the audit results and stated they would be 
rewriting and updating AFI 15-114 to establish standardized procedures for 
obtaining mission operator feedback on weather forecasts and identifying 
improvement opportunities within the MEFVER program, with an estimated 
completion date of 1 March 2006. The AF/A3 also agreed to amend existing 
processes to require operators to provide formal feedback on weather forecast 
accuracy and mission impact with an estimated completion date of 1 November 
2005. 

Some of the problems cited in the audit report could be fixed if CWTs were 
fully integrated into the flying squadrons they support. AFW does not have the 
manpower to this, so other fixes must be found. In our study we developed a 
potential solution to the problem of Air Force personnel not consistently and 
completely obtaining customer feedback and identifying improvement 
opportunities. The audit recommended AFW establish standardized procedures 
for obtaining mission operator feedback on weather forecasts. Further, the audit 
directed Air Staff agencies to coordinate and amend existing processes to 
require operators to provide formal feedback on weather forecast accuracy and 
mission impact (AFAA 2005). Our research project provides a process for 
meeting these recommendations and directives. 
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6. Air Force Weather Strategic Pian and Vision 

The AFW Strategic Plan and Vision (AFWSPV) issued August 2004 
(AFWSPV 2004) was intended to set AFW’s course for transformation starting 
with fiscal year 2008. The plan outlines eight “transformation vectors” that guide 
the weather core processes of collection, analysis, prediction, tailoring, and 
dissemination. The eight vectors are: 

1) Integrate environmental information collection into the theater 
intelligence, surveillance, and reconnaissance collection plan. 

2) Anticipate and manage the data explosion from the next generation 
environmental satellites, unmanned vehicles, national technical 
means, smart tankers, future combat system vehicles, etc. 

3) Advocate space weather models going from climatologically and 
statistically based to physics based. 

4) Anticipate and manage increasing model resolution, vertical domain 
from surface to near space, and physics requirements based on 
new weapon systems coming into the inventory. 

5) Move from a “graphically based” to “digitally based” product line. 

6) Exploit automated decision tools as a mainstream product (some 
will be machine-to-machine (M2M), others will be semi-automated). 

7) Fully integrate weather information into decision cycles at the 
appropriate levels of command. 

8) Fully embed people with users as “weather/climate consultants” 

In our study, we have addressed the last three vectors, especially in terms 
of exploiting automation (vector 6), determining where weather information is 
integrated into aviator decision cycles, from the base level down (vector 7), and 
the value of embedded weather personnel in flying squadrons (vector 8). 

AFWSPV (2004) states, “if you ‘boiled down’ the AFW business to its 
essence, you would find that we integrate into operations and intelligence, that 
our analyses provide battlefield situational awareness, and our predictions and 
tailored products enhance decision superiority for commanders at every level” 
(AFWSPV 2004). Our study provides a prototype system for enabling improved: 
(1) integration of AFW personnel into operations; (2) CWT awareness of how 
aviators view and use AFW products to plan and execute their missions; and (3) 
tailoring of products to improve customer decision making. 
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B. PREVIOUS WORK 

1. Metrics of Forecast Performance and Impacts on War Fighting 
Operations 

In 2005, Captain Jeff Jarry, USAF, (Jarry 2005) performed anaiyses of 
MEFVER data for FY 2004 provided by the Air Mobiiity Command (AMC). The 
three main motives behind his study were to; 

1) determine if AFW personnei within AMC were adding vaiue to war 
fighting operations 

2) improve the coiiection and anaiysis of AFW MEFVER data 

3) provide constructive inputs for the review and update process for 
the Air Force reguiations that govern metrics verification programs 
and procedures 

Jarry (2005) analyzed MEFVER data from 13 CWTs. However, some of 
the CWTs did not provide data on all of the MEFs they issued and it was not 
clear how many missions were actually supported by the reported MEFs. Jarry’s 
results clearly indicated a need for a more automated and near-real time system 
for collection and analyses of data. 

Jarry (2005) also found that “the main return on investment in the CWTs 
probably comes from their forecasts of relatively uncommon, but mission critical. 
No Go conditions”. A No Go forecast is issued when one or more phenomena 
are forecasted to be below thresholds as determined by weather personnel. 
Weather personnel receive the thresholds from their customers. 

Jarry (2005) points out that MEFVER data collection needs to be done in a 
more consistent and complete way. Further, he states that, “(CWTs) also need 
to record for each MEF how operators used the forecasts and the successes and 
failures of the corresponding mission. If this data were collected and 
appropriately archived and analyzed, AFW would be in a much better position to 
objectively and quantitatively demonstrate both its performance and the value it 
provides to operations, and to do so on a near-real time basis.” In our study we 
have developed and tested a system for meeting this need. 

2. System for Conducting Quantitative Near Reai Time Analyses 

In September 2005, Lieutenant Commander Mark Butler, USN, completed 

a project designed to develop and operationally implement an online system for 
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collecting and analyzing data on the performance and operational impacts of 
forecasts, and then reporting the results of the analyses in near real time (Butler 
2005). The results of this project were: 

1) identification of the data necessary for quantitatively determining 
forecast performance and the operational impacts of those 
forecasts 

2) development of an online data collection system linked to an online 
database for archiving and analyzing the required data into a 
readily accessible form 

3) development of a web-based interface that would provide users 
with near real-time access to the metrics results (i.e., the results of 
the data analyses) 

4) development of a final metrics report format that fully supported the 
needs of the users (i.e., Naval meteorology and oceanography 
(METOC) and AFW units and their operational customers) 

The system was tested and operationally implemented at the Naval Strike 
and Air Warfare Center at Naval Air Station Fallon. The system has been 
deemed quite effective by the METOC officer in charge of Naval Pacific 
Meteorology and Oceanography Detachment (hereafter referred to as NPMOD) 
at NAS Fallon (Cantu 2005). NPMOD provides METOC support to Naval strike 
airwings preparing for upcoming deployments. A major feature of the system is 
its ability to assess the operational impacts of METOC support, a process that at 
the time of the Butler study did not exist in any other civilian or military 
organization (Butler 2005). 

C. GOALS OF THIS STUDY 

We designed our study to address the directives, requirements, 
recommendations, needs, and challenges described in the preceding sections of 
this chapter. We focused in particular on developing methods for improving the 
collection and analysis of aviator feedback on the forecasts they received in 
support of their missions. Using this feedback, we calculated metrics of the 
performance and operational impacts of these forecasts, from the perspective of 
the aviation customers. We chose to focus on weather support for air combat 
operations in two major commands. Air Combat Command (ACC) and Pacific Air 
Forces (PACAF). Working with these major commands, gave us the opportunity 
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to assess weather support for a wide range of platforms, including fighters, 
bombers, and unmanned aerial vehicles. 

Our first goal was to adapt the Butler (2005) web-based, near-real time 
system for collecting and analyzing AFW, ACC, and PACAF data in order to 
assess: 

1) forecast performance 

2) weather phenomena with operational impacts 

3) operational impacts of those forecasts and phenomena. 

Our emphasis was on the third of these assessments. 

Our second goal was to implement and test our system with ACC and 
PACAF CWTs. Our objective was to create a highly streamlined and automated 
system so that there would be little or no additional work for the CWTs and their 
customers to do. 

Our third goal was to use the results from system implementation to 
provide AFW, ACC, and PACAF personnel with advice on how to alter 
procedures and allocation of resources to better accomplish their goals. Our 
objectives were to calculate and provide forecast performance and operational 
impacts metrics, improve terminal forecast reference notebooks, provide advice 
on the alteration of MEFs and instrument refresher courses, and provide near- 
real time results in a format ready for use by leadership. 

Our intention was to provide improved tools for CWTs to use in meeting 
the requirements described in AFMAN 15-129 and AFI 15-114, and for 
leadership to use in addressing the findings in the AFAA (2005) report. We also 
set out in this study to make progress along three of the AFWSPV vectors, 6-8 — 
in particular by helping to automate CWT processes, quantifying the use and 
impacts of weather information in customer decision making processes, 
assessing the value of embedded CWT personnel, and minimizing the impacts 
of, or reducing the need for, full integration. 
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II. DATA AND METHODS 


A. DATA COLLECTION METHODS 

1. Initiation of Data Coliection Efforts 

Our data collection efforts were focused on fighter aircraft for several 
reasons. First, fighters represent a significant AF weapon system. Second, 
fighters were chosen to expand on other studies, such as Jarry (2005) which 
covered AMC air transport aircraft. Finally, the majority of my weather 
experience has been in support of fighter aircraft. 

To initiate our collection efforts, we contacted the standardization and 
evaluation branch in the weather division at headquarters ACC to obtain data 
from CWTs supporting ACC assets. ACC is the primary provider of combat 
airpower to U.S. war fighting commands. To support global implementation of 
national security strategy, ACC operates fighter, bomber, reconnaissance, battle- 
management, and electronic-combat aircraft. ACC also provides command, 
control, communications and intelligence systems, and conducts global 
information operations. 

The ACC web site (ACC 2006) explains their mission as a force provider, 
organizing, training, equipping, and maintaining combat-ready forces for rapid 
deployment and employment to meet the challenges of peacetime air sovereignty 
and wartime air defense. ACC numbered air forces provide air components to 
U.S. Central and Southern Commands, with Headquarters ACC serving as the 
air component to U.S. Northern and Joint Forces Commands. ACC also 
augments U.S. European, Pacific and Strategic Commands forces (ACC 2006). 

To support our project, the ACC director of weather contacted all the ACC 
CWTs encouraging support of our project. Five CWTs agreed to collect data for 
our project. These CWTs support a variety of ACC aircraft, including A-10s, B- 
1s, B-2s, F-16s, and U-2s. Three of the CWTs were involved in the AFAA 
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metrics audit (AFAA 2005). This is important to note because our project was 
designed to provide a system for assisting CWTs in addressing some of the 
deficiencies noted in the audit. 

After coordination and initiation of data collection with the ACC CWTs, we 
expanded our data to include two CWTs in PACAF. PACAF’s primary mission is 
to provide ready air and space power to promote U.S. interests in the Asia- 
Pacific region during peacetime, through crisis, and in war (PACAF 2006). 
CWTs in PACAF have fully integrated weather personnel into the flying 
squadrons. This gave us an opportunity to collect data that could be used to 
assess the value added by embedding weather personnel. We focused our 
efforts on collecting data from two PACAF units in Korea because they are more 
operational and fly more missions than CONUS units. 

2. Know Your Customer 

Our process of coordinating with the ACC CWTs started with us providing 
guidance on building customer relationships and collecting customer feedback. 
This section and the following section describe the major steps we recommended 
be taken by the CWTs to improve their knowledge of, and interaction with, their 
customers. Our objective was to facilitate the participation, and increase the 
effectiveness, of the CWTs in the customer feedback portion of our project. 
a. Understand Supported Mission and Aircraft 
The first step is developing a firm understanding of the supported 
mission and aircraft. Without this understanding, collection of useful data will be 
difficult. This involves obtaining weather thresholds defined in the weather 
support plan. After reviewing the plan, one must visit the supported flying 
squadrons and obtain weather sensitivities from squadron leadership, planners, 
and safety officers. This information then needs to be combined with weather 
threshold data to develop a basis for tailored forecasts. 

Good references to use in the process of understanding supported 
aircraft are 11-series AFIs covering flying operations. AFI 11-214, Air Operations 
Rules and Procedures, 2001 (hereafter referred to as AFI 11-214), provides rules 

and procedures for air-to-air and air-to-surface operations and training. The AFIs 
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give overarching guidance that is often supplemented by MAJCOMs and Air 
Force installations. These supplements will never be more lenient when it comes 
to thresholds, but may be more restrictive, so it is necessary to review all 
guidance from the APIs and the supplements, from the MAJCOM supplements 
right down to those from the base, group, and squadron. 

An excellent location to find weather limitations for specific aircraft 
is in a document called the Operational Weather Limiters Search (OWL). The 
OWL can be found on the website of the 15**^ Operational Weather Squadron 
(OWL 2006). This is not an official Air Force document but can still be helpful for 
general guidance and is a good, easy to read, starting point for aircraft 
thresholds. Once the general guidance is procured, interviews with 
standardization and evaluation staff in the operations group can help firm up the 
limitations and thresholds for the specific aircraft supported. In addition, the 
airfield operations staff will be able to provide takeoff and landing criteria for the 
airfield. 

Finally, weather personnel must have a firm understanding of the 
aircrew they support. There are differing takeoff and landing criteria based on an 
aircrew’s experience level. It is important to know those criteria so the thresholds 
can be incorporated into the forecasts. The flying squadron director of 
operations can be interviewed to gain knowledge about aircrew in upgrade 
training, average number of days per week individuals fly, crew rest 
requirements, and overall demands placed on flying customers. Integration with 
the supported unit is important. 

b. Develop a Timeline of How Business is Conducted 
Once the customer is known, a basic timeline of the customer’s 
planning, execution, and debriefing process needs to be defined. This timeline 
should start at the point where decisions are first made at the particular 
installation. The wing scheduling office is a good place to start because this 
office typically coordinates the basic flying schedule for the entire wing. Once the 
general flow of aviation operations is understood, then a general weather support 

timeline can be developed. Figure 1 shows a schematic operational timeline with 
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decision points at which weather products can be injected into the decision 
making process. Long range planning, such as the scheduling of exercises and 
deployments by wing offices, will use climatological weather products, whereas 
an aircrew planning a flight they will execute within three hours will use a MEF. 

Specific customer requirements are very important to know to 
determine times when weather support is needed, and what type of support is 
required (e.g. cloud ceilings, surface visibility, satellite image, thermal crossover 
data), and what format should be used (e.g. electronic copy, face-to-face 
briefing). This information is critical in order to provide the support the customer 
needs to make decisions on tactics, weapon type, and scheduling. 

3. Design of Data Collection Process 

A firm understanding of the customer’s processes is also critical to the 
collection of the data needed to assess forecast performance and the operational 
impacts of forecasts. For example, knowledge of the points within the customer’s 
processes at which weather information is used by the customer is critical for 
determining what data needs to be collected on the weather information provided 
to the customer and the decisions made by the customer based on that 
information. 

Figure 2 shows a schematic flow chart for planning, executing, and 
debriefing air operations (left column), and the corresponding data collection 
needed to analyze the performance and operational impacts of the weather 
support provided for the operations (right column). The chart starts at the wing 
level and works its way down to the mission execution and debrief of the mission. 
The mission debrief is crucial because this is where weather personnel can learn 
how the weather products they provided, and the weather phenomena actually 
encountered, impacted the mission. Information from debriefs is a major part of 
the foundation for improving weather support to aviators, and a focal point in our 
data collection process. Much of our study focused on developing methods for 
efficiently collecting information from aviators on how the forecasts and the actual 
weather encountered affected their missions. 
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In order to determine the types of data that needed to be collected, we 
developed flow charts depicting all the weather related steps in the mission, from 
the receipt of a planning weather forecast all the way to a mission debrief. These 
flow charts describe three main events in mission planning, execution, and post¬ 
mission assessment: 

1) Once a planning weather forecast is received by an aviator, a 
determination is made by the aviator as to whether or not the 
mission plan should be changed. 

2) The aviator then receives the mission execution forecast and once 
again determines whether or not the mission plan should be 
changed. 

3) Unless the crew decides to cancel the mission, it is flown and the 
crew determines whether or not weather negatively impacted the 
mission and whether the mission was successful or unsuccessful. 

In developing our flow charts, we found it helpful to work backward from 
an end result of either a successful mission or an unsuccessful mission. Figure 3 
shows all the ways a successful mission can result from: (1) a planning weather 
forecast (PWF) indicating a negative mission impact; and (2) a PWF indicating no 
negative mission impact. Figure 4 shows all the ways an unsuccessful mission 
can result from both of these types of PWFs. The process is essentially the 
same as for successful missions. For each figure, there are 16 paths shown by 
arrows. These paths represent the different types of weather information that 
could be provided to an aircrew, the different types of decisions the aircrew could 
make in response to that information, the different impacts the actual weather 
could have on the mission, and the different types of mission outcome. 

Our data collection process was designed to collect the data needed to 
determine, for each mission, which of these paths was followed. Our data 
analysis process was designed to determine how and to what extent weather 
forecasts, and the weather phenomena encountered during the mission, affected 
aviator decisions and mission success. 
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Once we determined what data needed to be collected, we developed a 
generic data collection form to be completed by aircrews. Figure 5 shows this 
form, which is broken into three main sections: operational impacts, forecast 
performance, and target acquisition weapons software (TAWS) data. TAWS 
data encompasses both operational and forecast performance information, 
although not all aircraft use TAWS. 

Note that the data collected via this form is focused on weather 
phenomena that were expected to have, and/or did have negative impacts on the 
mission. This is because we chose to limit the scope of our project to the 
phenomena that are most difficult to forecast and that have the most potential for 
altering mission plans and hindering mission execution, which are the negatively 
impacting phenomena (Hinz 2004, Jarry 2005, Butler 2005). 

Note also that this form collects data solely from aviators. Thus, the data 
collected via this form represents aviator reports of; (1) negative weather impacts 
the aviators inferred from the forecasts they received; (2) the changes the 
aviators made to their mission plans in response to the negative impacts they 
inferred from the forecasts; (3) the weather phenomena encountered by the 
aviators during their missions; and (4) the changes made by the aviators to their 
missions due to the negatively impacting weather phenomena they encountered; 
and (5) the degree of success of their missions. 

We had two main motivations for collecting this data solely from the 
aviators. First, we wanted to develop and test a process for meeting the 
requirements imposed on CWTs to collect and evaluate customer feedback. 
Second, the aviators commonly make their own inferences from the forecasts 
about negative weather impacts, and do not rely on the negative impacts that 
may or may not be provided by the CWTs. Thus, In effect, the aviators are 
acting as forecasters. They are at least acting as forecasters of the negative 
impacts, and perhaps also of the weather itself, if they do not assume the 
forecasts are accurate (i.e., if they do not believe the forecasts). Thus, we felt it 
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was important to assess the performance and operational impacts of the aviators’ 
forecasts. These issues are discussed further in the data analysis section 
(section F) below. 

Figure 6 shows a separate generic data collection form developed to gain 
feedback on a relatively new program called infrared target scene simulation 
(I RTSS). I RTSS software creates a three-dimensional picture of what the target 
area will look like under specific environmental conditions. IRTSS has not yet 
been fielded throughout the Air Force, but is being used by at least one of the 
bases participating in this study, as part of the military’s evolution toward effects- 
based operations (AFWSPV 2004). The IRTSS data collection form is relatively 
basic because the IRTSS program is still in its early stages. 

B. CWT DATA COLLECTION 

The generic data collection form was sent to the participating CWTs with a 
request to tailor the form to reflect their specific customers’ missions. When 
drafting answer options for each question, the CWTs were directed to offer 
enough options to cover what they anticipated to be 80 percent of their 
customers’ likely responses. The different CWTs support very different missions, 
which proved to be challenging in developing a relatively uniform data collection 
and analysis process. 

1. Beale AFB 

a. Supported Missions 

The Beale AFB web site (Beale 2006) provides the following 
description of the mission of the 9**^ Reconnaissance Wing: “[It] is responsible for 
providing national and theater command authorities with timely, reliable, high- 
quality, high-altitude reconnaissance products. To accomplish this mission, the 
wing is equipped with the nation’s fleet of U-2 and RQ-4 reconnaissance aircraft 
and associated support equipment”. There are also T-38 and reserve KC-135 
aircraft stationed at Beale AFB, California. 

The U-2 provides high-altitude, all-weather surveillance and 
reconnaissance, day or night, in direct support of U.S. and allied forces. It 
delivers critical imagery and signals intelligence to decision makers throughout all 
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phases of conflict, including peacetime indications and warnings, low-intensity 
conflict, and large-scale hostilities (Factsheets 2006). 


The Beale CWT collected data for U-2 operations because of the 
U-2’s unique missions and thresholds, and because U-2 units have a more 
formal debrief process which increased collection opportunities. The U-2 
performs both “high” and “low” altitude flights. The low flights receive their MEF 
via a computer terminal in the squadron, but the high flights actually go to the 
weather station to receive their MEF. Because of the direct interaction with the 
aircrews, data was collected on the high flights. The generic data collection form 
(Figure 5) was only slightly tailored for the U-2 high flights. 
b. Data Collection Methods 

The Beale CWT attempted to collect data in November and 
December 2005, but had little success in getting pilots to return the forms. The 
data collection forms were being handed to the aviators when they received their 
briefing at the weather station with the request that they be filled out and returned 
after the mission. After little success in getting pilots to fill out and return the 
forms, the CWT emailed the forms to the pilots during an intensive data collection 
period on 9-13 January 2006 and was more successful in getting completed 
forms from the pilots. 

2. Creech AFB 

a. Supported Missions 

The mission at Creech AFB, Nevada has evolved significantly in 
the past year (Creech 2005). In March 2005, the Air Force unmanned aerial 
vehicle (UAF) center of excellence (COE) was stood up, and on 20 June 2005, 
the Indian Springs Air Force auxiliary air field was renamed Creech AFB in honor 
of General Wilbur L. “Bill” Creech. October 2005 saw the standup of the Joint 
UAV COE at Creech AFB (Creech 2005). 

Creech AFB is home to the 11**^ and 15*^ Reconnaissance 
Squadrons (Creech 2006), whose primary mission is to provide theater 
commander in chiefs (CINCs) with deployable, long endurance, real-time aerial 
reconnaissance, surveillance, and target acquisition flying the RQ-1A Predator. 
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The Predator reports battlefield conditions to war fighters using medium-altitude, 
multi-sensor, platforms and also collects and distributes imagery products to 
CINCs and national level leadership (Creech 2006). Additionally, the 
Reconnaissance Squadron conducts all Predator aircrew qualification training 
(Creech 2006). The generic data collection form (Figure 5) was not changed for 
Creech AFB because the CWT had difficulty coordinating the form with aviators. 
b. Data Collection Methods 

The weather flight commander at Creech AFB found little to no time 
to coordinate the generic data collection form and therefore was not able to 
develop a tailored data collection form and implement a data collection process. 
Thus, we did not collect data from Creech AFB. 

3. Dyess AFB 

a. Supported Missions 

The Dyess AFB web site (Dyess 2006) explains the mission of the 
7*^ Bomb Wing (BW) as follows. The 7*^ BW is the premier operational B-1B unit 
in the U.S. Air Force. It has the capability to project lethal airpower anywhere in 
the world and strike at a moment’s notice. During Operation DESERT FOX, a 
Dyess B-1 matched with a B-1 from Ellsworth Air Force Base, S.D., bombed Iraqi 
Republican Guard barracks at Al-Kut, Iraq, with 500-pound MK-82s (Dyess 
2006). This first-ever combat bomb drop displayed the destructive firepower of 
the B-1 and echoed the battle cry “MORS AB ALTO,” Latin for Death From 
Above (Dyess 2006). The B-1 carries the largest payload of both guided and 
unguided weapons in the Air Force inventory and is the backbone of America’s 
long-range bomber force. It can rapidly deliver massive quantities of precision 
and non-precision weapons against any adversary, anywhere in the world, at any 
time (Factsheets 2006). 

There are six flying squadrons located at Dyess AFB, Texas; four 
B-1 squadrons and two C-130 squadrons, although the CWT only coordinated 
data collection with the B-ls because of their unique mission. The generic data 
collection form (Figure 5) was slightly altered to adapt to the B-1 mission. Initial 
data collection revealed the need to adjust wording in the data collection form to 
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make it more understandable to aviators. Data collection was started in 
November and continued through the intensive data collection period of 9-13 
January 2006. 

b. Data Collection Methods 

The CWT at Dyess AFB coordinated data collection with the 
operations officers at the flying squadrons. The flying squadrons agreed to 
maintain blank data collection forms at the squadron operations desk, since this 
is one of the locations at which aviators fill out paperwork upon completion of 
their missions. CWT personnel entered data into electronic forms and forwarded 
them to NPS for data analyses. 

4. Nellis AFB 

a. Supported Missions 

Several different types of operations occur simultaneously at Nellis 
AFB, Nevada. We focused data collection efforts on the USAF weapons school 
(USAFWS), air warrior, and red flag because their schedule is regimented and 
predictable. The mission of the USAFWS (Nellis 2006) is to teach graduate-level 
instructor courses, which provide the world’s most advanced training in weapons 
and tactics employment to officers of the combat air forces. Aircraft flown at the 
USAFWS include A-10, AC-130, B-1, B-2, B-52, F-15C, F-15E, F-16C, F-117, 
HH-60, MC-130, and MH-53 (Nellis 2006). 

Air warrior and red flag are exercises conducted at the 414**^ 
combat training squadron (Nellis 2006) using realistic training scenarios that are 
intended to develop the combat readiness and survivability of participants. 
Combat units from the U.S. and several allied countries engage in combat 
training scenarios conducted within the Nellis Range Complex (Nellis 2006). 

The USAFWS schedule encompasses both academic and flying 
phases. The prime opportunity for data collection is during the flying, or mission 
execution (ME), phase. The ME phase lasts one week with several flights each 
day. The generic data collection form (Figure 5), was only slightly adjusted to 
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accommodate USAFWS missions. TAWS is heavily used at the USAFWS as 
well as IRTSS, so the IRTSS questionnaire in Figure 6 was also used at Nellis 
AFB. 

b. Data collection Methods 

Data collection at the USAFWS was coordinated with a squadron 
commander at the school who forwarded all the aviators an electronic copy of the 
data collection form requesting/directing support. Completed forms were sent to 
NPS for analyses. 

Data collection forms were also provided to weather personnel 
supporting both air warrior and red flag. These weather personnel were not 
stationed at Nellis AFB, but rather on temporary duty (TDY) for the purpose of 
providing weather support to the aviators participating in the two exercises. 
Aviators were provided paper copies of data collection forms prior to mission 
execution and completed forms were sent to NPS for analyses. 

5. Whiteman AFB 

a. Supported Missions 

Whiteman AFB, Missouri is the home of the 509**^ Bomb Wing, 
which according to their web site (Whiteman 2006) operates and maintains the 
Air Force’s premier weapon system, the B-2 bomber. The B-2 Spirit is a multi¬ 
role bomber capable of delivering both conventional and nuclear munitions. A 
“dramatic leap forward in technology” (Factsheets 2006), the bomber represents 
a major milestone in the U.S. bomber modernization program. The B-2 brings 
“massive firepower to bear, in a short time, anywhere on the globe through 
previously impenetrable defenses” (Factsheets 2006). 

Two bombing squadrons are located at Whiteman AFB and data 
was collected from both. The generic data collection form (Figure 5) was 
significantly shortened for use at Whiteman AFB because the aviators place the 
majority of their emphasis on the MEF with little on the planning weather 
forecasts (PWFs). 
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b. Data Collection Methods 

PWFs are provided to aircrews telephonically and via the web, 
although MEFs are briefed in person approximately three hours prior to takeoff. 
The Whiteman AFB CWT just recently integrated weather personnel into the 
flying squadrons, in part due to classification issues with the B-2. Classification 
issues also result in restrictions to CWT access at locations in which the aviators 
perform debriefs. Thus, paper copies of the data collection form were placed in 
the debrief area at the maintenance operations squadron and then faxed to the 
CWT upon completion. 

6. PACAF Units 

a. Supported Missions 

The two PACAF bases from which we collected data were Kunsan 
Air Base (AB) and Osan AB, both located in South Korea. CWT personnel are 
well integrated into the flying squadrons, which is why these bases were asked to 
participate. 

Kunsan AB is located on the western side of the South Korean 
peninsula bordered by the Yellow Sea, approximately 150 miles south of Seoul. 
Kunsan AB is home to the 8*^ Fighter Wing made up of two F-16 fighter 
squadrons (Kunsan 2006). The F-16 Fighting Falcon is a compact, multi-role, 
highly maneuverable fighter aircraft which has proven itself in air-to-air combat 
and air-to-surface attacks (Factsheets 2006). It provides a relatively low-cost, 
high-performance weapon system for the U.S. and allied nations (Factsheets 
2006). 

Osan AB is located just 48 miles south of the Korean DMZ and is 
home to the 51 Fighter Wing “Mustangs” (Osan 2006). It is the most forward 
deployed permanently-based wing in the Air Force and provides mission ready 
airmen to execute combat operations and receive follow-on forces. The wing has 
24 PAA, F-16, and A-10 squadrons, along with a C-12 airlift flight, conducting a 
full spectrum of missions in defense of the Republic of Korea (Osan 2006). The 
A/OA-10 Thunderbolt II (Factsheets 2006) is the first Air Force aircraft specially 

designed for close air support of ground forces. The aircraft are a simple, 

26 



effective, and survivable twin-engine jet that can be used against all ground 
targets, including tanks and other armored vehicles (Factsheets 2006). 

b. Data Collection Methods 

The CWTs at Kunsan AB and Osan AB did not have the time to 
coordinate applicable questions and answer options on the data collection form, 
so the Nellis AFB data collection form, including the TAWS section, was used. 

CWT personnel are fully integrated into the flying squadrons at 
Kunsan AB, available four to five hours prior to takeoff time and remaining until 
the last aviator steps out the door to fly. Paper copies of the data collection form 
were completed by flight leads on behalf of all the aircraft in their flight, consisting 
of either a 2-ship or 4-ship formation. Completed forms were mailed to NPS for 
analyses. 

At Osan AB, one CWT person is embedded in each of the flying 
squadrons, providing mass briefings three hours prior to takeoff and updates 
(step briefings) one hour prior to takeoff. The Osan AB CWT was already 
collecting feedback from flight leads in the form of a paper copy, so our data 
collection form was used in place of the existing one. CWT personnel briefed 
flying squadron leadership on the purpose and importance of the form to help 
ensure data quality and quantity. Completed forms were mailed to NPS for 
analyses. 

C. NPS AVIATOR SURVEY 

Initial feedback from the ACC and PACAF CWTs indicated the data 
collection form was too lengthy, which might negatively affect the number of 
forms that were completed, and the quality of the data on the forms that were 
completed. To improve the forms, we surveyed AF pilots and navigators 
attending NPS. We also conducted subsequent follow up surveys and interviews 
with the NPS aviators to validate and interpret the results from our analyses of 
the data collected from the ACC and PACAF aviators. 

Although most NPS aviators have only been in a non-flying job for a short 
time, being in the academic environment has provided them the opportunity to 
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reflect on their experiences. Their NPS experiences have perhaps also given 
them a motivation for contributing to our research project, since they too were 
collecting and analyzing data for their thesis research projects. Figure 7 is the 
survey emailed to the 36 AF aviators at NPS, most of who have attained the rank 
of major, have at least nine years of active duty AF experience, and several 
hundred hours of flying experience. 

D. SUMMARY OF DATA COLLECTED 

1. Aircrafts and Missions 

Data was collected from six AF bases on six different types of aircraft. 
Figure 8 gives a summary of the missions flown during the data collection 
timeframe, including average number of missions flown per day, average length 
of each mission, and the type of mission (e.g., air-to-air, air-to-ground, bombing). 
A significant number of the missions were from fighter aircraft, perhaps because 
fighter missions are shorter than bombing and reconnaissance missions and 
therefore more frequent. 

2. Data 

Data collection was initiated in November 2005 and ended in January 
2006, with the objective of having data collection forms completed by aviators, 
not CWT personnel. Concentration was placed on collected data from all the 
bases on five consecutive days, 9-13 January 2006, and for no less than five 
consecutive hours each day (and preferably for the entire day). We did not 
attempt to collect the same percentage of data from each base due to differences 
in flight operations and CWT manning. 

E. LIMITATIONS OF DATA COLLECTION PROCESS 

1. Assumptions 

We made several assumptions regarding or data collection process. First 
we assumed that the CWTs were provided sufficient guidance on how to get to 
know their customer prior to coordinating their data collection forms. Second, we 
assumed that the CWTs were successful at tailoring their data collection forms to 
develop mission appropriate questions and answer options. Third, we assumed 
those individuals filling out the data collection forms fully understood the 
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questions (especially after we had altered the initial forms in response to aviator 
and CWT feedback). Fourth, we assumed the aviators providing the data would 
provided relatively accurate descriptions and assessments of the forecasts they 
received, the missions changes they made in response to the forecasts, the 
weather they encountered during their missions, and the missions changes they 
made in response to the weather they encountered, 

2. Limitations 

A major shortcoming was the limited amount of data collected due to the 
limited time we had to collect data. This was exacerbated by the extended 
periods of holiday leave and down time that occurred during our data collection 
period. We also relied on aviators to provide accurate assessments of weather 
forecasts, weather phenomena encountered, planning changes, and weather 
impacts during execution. The quality of the data they provided, especially the 
weather data, is difficult to determine, since we did not, and in general were not 
able to, verify the aviator data with independently collected data. In some cases 
security limitations, especially at Whiteman AFB, restricted our ability to collect 
data. 

F. DATA ANALYSIS METHODOLOGY 

1. Overview 

Hinz (2004), Jarry (2005), and Butler (2005) used a method in which they 
collected weather and operator data, used that data to determine forecast and 
operator performance metrics, and then compared these two sets of metrics to 
determine operational impacts metrics. This method is schematically depicted in 
Figure 9 in the form of a flow chart showing the data collected on the top row, 
performance analyses in the middle row, and the assessment of operational 
impacts on the bottom row. For our project, all the data was provided by 
aviators. We then analyzed the aviator data to assess forecast performance, 
aviator performance, and the impacts of the forecast and the actual weather 
encountered on the operational performance of the aviators. The results of our 
analyses are metrics of forecast and operational performance, and the 
operational impacts of the forecasts and actual weather phenomena. 
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In prior studies, such as Hinz (2004), Jarry (2005), and Butler (2005), data 
was collected data from weather personnel, and analyzed by weather personnel. 
We chose to work with data obtained directly from customers because we 
wanted to get and assess the value of direct customer feedback. 

2. Goals 

The focus of our project was not on collecting and analyzing large 
quantities of data, but rather developing a process for collection and analyses. 
We did not attempt to come up with thorough analyses of forecast and 
operational performance, or of the operational impacts of forecasts and weather 
phenomena encountered during mission execution. Our goal was to design, 
develop, and test a new process for collecting and analyzing customer feedback. 
The deliverable product we set out to create was a prototype system capable of 
efficiently collecting and analyzing customer data for use in analyzing forecast 
and operational performance, and the operational impacts of forecasts and 
weather phenomena encountered during mission execution. 

To develop such a prototype system, we created system components to 
perform the following functions. From an operational perspective, a primary 
system function is to determine what operational decisions were being made 
based on PWFs and MEFs. We also wanted to determine key timeframes in 
which aviators use weather forecasts, and whether or not aviators felt forecasts 
were available when they needed them. 

From a meteorological perspective, a primary system function is to 
determine how accurate the weather forecasts were, the probability of detecting 
negatively impacting weather phenomena, and the subsequent false alarm rate 
for negatively impacting weather phenomena. Previous studies determined 
these parameters from the perspective of weather personnel. Our objective was 
to determine them from the perspective of aviators. We also wanted to 
determine how believable the weather forecasts were to the aviators. In addition, 
we wanted to identify the weather phenomena with negative operational impacts 
that were most commonly misforecasted, and those that most commonly resulted 
in unsuccessful missions. 
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Overall, our main goal was to develop a prototype system for analyzing a 
combination of operational and weather performance data provided by aviators in 
order to assess the impacts of forecasts and weather phenomena on the 
planning and execution of air combat missions. One of our main motivations was 
to enable CWTs to satisfy several requirements in AFMAN 15-129, AFI 15-114, 
the AFAA report, and the AFWSPV. These requirements include the 
development of a user-friendly system for measuring the operational impacts of 
weather forecasts. Once our prototype is tested, it will be adaptable to any CWT 
in AFW. Our prototype will also be adaptable to Naval METOC units to assist in 
their assessment of the operational impacts of METOC products. 

An additional goal was to support the developers and implementers of 
TAWS and IRTSS by collecting and analyzing TAWS and IRTSS data. New 
versions of TAWS continue to be fielded and because of feedback, the versions 
get better and more accurate. Nellis AFB is the primary active duty AFB using 
IRTSS and since Nellis agreed to participate in this project, IRTSS developers 
requested our support in obtaining operational feedback. 

3. Metrics Calculated 

The main results of our analyses were five quantitative metrics of forecast 
performance (items a-e, below), and 15 quantitative metrics of operational 
impacts (items, f-t, below). Several of these metrics, especially those of forecast 
performance, have been used in prior studies (e.g., Hinz 2004, Jarry 2005, Butler 
2005). However, our calculation of these metrics was based on a different type 
of data than that used in prior studies. The main difference is that our data: (1) 
came entirely from the users of the forecasts, the aviators, not from weather 
personnel; and (2) focused exclusively on weather phenomena that was 
forecasted to and/or did have negative impacts on the missions. Despite these 
differences, comparisons of the two types of metrics are useful, and are 
discussed in chapter III. 

For our study, we focused on negatively impacting phenomena in the 

forecasts and in the pilot report observations. Thus, our forecast performance 

metrics are comparable to, for example, red forecast accuracy metric of Hinz 
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(2004) and the no go forecast accuracy of Jarry (2005). In our case, forecast 
performance metrics are based on the aviators inferring from the forecasts the 
weather phenomena that they think will cause negative impacts, and then 
verifying that inference when they execute the mission. In a sense, this means 
that we treat the aviators as forecasters because we rely on them to infer the 
phenomena that will have negative impacts, and also as verifiers of the forecasts 
because they provide the observations of the actual weather. Before we 
calculated forecast performance metrics based on the aviator data, we checked 
that the time and location of the forecasts matched the time and location of the 
missions. Only the data from missions with matching times and locations were 
used in the calculation of those metrics. 

a. Forecast Accuracy for Negatively Impacting Weather 
Phenomena 

Forecast accuracy (FAC) is the percentage of forecasts that were 
accurate (Hinz 2004, Jarry 2005). For our study, FAC is the number of missions 
for which negatively impacting weather was inferred, and actually occurred, by 
the aviators, divided by the number of missions for which negatively impacting 
weather was inferred, all multiplied by 100 to come up with a percentage. 

b. Probability of Detection for Negatively Impacting 
Weather Phenomena 

Probability of detection (PCD) is defined as a verification measure 
of categorical forecast performance equal to the total number of correct event 
forecasts divided by the total number of events observed (NCAA 2006). For our 
study, PCD is the number of missions for which a specific negatively impacting 
phenomenon (e.g., cloud ceilings below thresholds) was inferred by the aviators 
from the forecasts and was also observed by the aviators, divided by the number 
of missions for which that negatively impacting phenomenon was inferred from 
the forecasts by the aviators, all times 100. 

c. False Alarm Rate for Negatively Impacting Weather 
Phenomena 

The NCAA verification glossary (NCAA 2006) defines the false 
alarm rate (FAR) as a verification measure of categorical forecast performance 
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equal to the number of false alarms divided by the total number of event 
forecasts. A false alarm is a forecast of particular weather phenomena that does 
not occur. In our case, if an aviator inferred from the forecast that cloud ceilings 
would have a negative mission impact, but did not actually experience this on the 
mission, then a false alarm for cloud ceilings occurred for that mission. We 
defined FAR as the number of missions for which a specific negatively impacting 
phenomenon was inferred from the forecasts by the aviators but did not actually 
occur, divided by the number of missions from which that negatively impacting 
phenomenon was inferred from the forecasts, all times 100. 

d. Observed Phenomena with Negative impacts That 
Were/Were Not Forecasted 

This metric describes the percentage of missions for which a 
specific negatively impacting phenomenon occurred during mission execution 
and whether the aviators inferred that phenomenon from the PWF and/or the 
MEF. There are four possibilities for this metric: (1) phenomenon occurred 
during execution, and was inferred from the PWF but was not inferred from the 
MEF; (2) phenomenon occurred during execution, and was not inferred from the 
PWF but was inferred from the MEF; (3) phenomenon occurred during execution, 
and was inferred from the PWF and was inferred from the MEF; (4) phenomenon 
occurred during execution, and was not inferred from the PWF and was not 
inferred from the MEF. This metric is somewhat similar to our FAC metric, but 
with a focus on specific phenomena, and without accounting for the total number 
of forecasts that were correct and incorrect. 

e. Percentage of Missions for Which Negative Mission 
impacts Occurred and Were/Were Not indicated by the 
MEF 

This metric describes the percent of missions that incurred negative 
weather impacts that were not inferred from the PWF or the MEF. This is similar 
to other metrics mentioned above but provides a more general assessment than 
the prior metrics. 
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f. Negative Mission impacts inferred from Pfenning 
Weather Forecast and MEF 

This metric describes the number and percentage of missions for 
which: (1) the aviators inferred from the PWF and the MEF that a specific 
negative impact would occur; (b) a specific negative impact occurred during 
execution. By specific negative impacts, we mean impacts such as scheduling 
changes, weapons changes, inability to takeoff, etc. 

g. Mission Pian Changes Made Due to Pfenning Weather 
Forecast and MEF 

This metric describes the number of missions for which aviators 
made mission plan changes base on negatively impacting weather phenomena 
being inferred from the planning weather forecast and/or MEF. 

h. Mission Pian Changes Due to Negative Mission impacts 

This metric describes the number of times a mission plan is 

changed due to aviators inferring negative mission impacts in forecasts. 

I. Weather Phenomena Resuiting in Unsuccessfui Mission 

This metric describes the percentage of missions deemed 
unsuccessful by aviators due to negative impacts from weather phenomena. 

j. Pfenning Weather Forecast Positive Mission 
Contribution 

This metric describes the percent of missions for which a positive 
mission impact (PMC) is made by the planning weather forecast. If aviators infer 
a negative mission impact from the planning weather forecast, make mission 
plan changes, and then achieve their primary mission, the PMC criteria are met. 

k. MEF Positive Mission Contribution 

This metric describes the percent of missions for which a positive 
mission impact (PMC) is made by the planning weather forecast. If aviators infer 
a negative mission impact from the planning weather forecast, make mission 
plan changes, and then achieve their primary mission, the PMC criteria are met 
for our study. 
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/. Potential Positive Mission Contribution 

This metric describes the percent of missions for which a positive 
mission contribution (PPMC) is made by the planning weather forecast or the 
MEF. Our study defines a PPMC as an accurate forecast is issued, aviators 
make no mission plan changes in response, and the primary mission is not 
accomplished. 

m. Planning Weather Delay Effects on Mission 
Accomplishment 

This metric describes the percent of unsuccessful missions where 
the aviator would have checked the planning weather forecast earlier had it been 
available. 

n. Negative MEF, No Mission Changes 

This metric describes the percent of missions that are successful 
even though negative weather impacts are forecasted in the MEF, but the aviator 
does not make any mission plan changes due to the MEF. 

o. Mission Plan Changes Due to Planning Weather and 
MEF 

This metric describes the number of times aviators make a mission 
plan change due to the planning weather forecast and then change the same 
mission plan after seeing the MEF. 

p. Unsuccessful Mission Due to Inaccurate MEF 

This metric describes the percent of missions deemed unsuccessful 
by aviators where the aviator feels the MEF is inaccurate. 

q. Weather Phenomena Actually Occurring Resulting in 
Mission Success/Failure 

This metric describes the percent of each weather phenomena 
actually occurring during the mission and then whether or not the aviators 
deemed the mission successful/unsuccessful. 

r. Planning Weather Needs 

This metric describes the percent of missions evaluated where the 
aviator would have checked the planning weather earlier had it been available. 
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s. Timeframe of MEF Usage 

This metric describes the number of hours prior to mission takeoff 
that aviators check the MEF. 

t. Successful Missions 

This metric describes the percent of missions considered 
successful by the aviator even though they did not accomplish what they 
intended to accomplish on the mission. 

Each different data collection form required a separate data 
analysis worksheet based on the initial coordination with aviators. For this 
reason, some the metrics listed above will not be calculated for every base. 
Figure 10 is a description of calculations performed to analyze ACC and PACAF 
data, plus instructions for displaying the results of the analyses. Figure 10 is 
based on data collected from Kunsan AB, Nellis AFB, and Osan AB. Data 
analyses for the other bases will be abbreviated versions of Figure 10 because 
data collection was not as robust. 

4. Online Data Collection and Analysis System 

a. Software 

To be most effective, assessment of forecast performance and the 
operational impacts of forecasts must be timely. Flinz (2004) and Jarry (2005) 
both had large lag times between their final collections of data from operational 
units until their final metrics reports were completed. Butler (2005) found that for 
a metrics program to be effective, the turn around time, from collection of data to 
delivery of metrics results, must be near-real time. This can be done by 
leveraging information technology (IT), which is just what Butler (2005) did. 

Butler (2005) used three computer languages. Hypertext Markup 
Language (HTML), PHP; Hypertext Preprocessor (PHP), and My Structured 
Query Language (MySQL), to allow users to enter data via the internet into a 
database located at NPS, have the data analyzed, and then receive results in 
near real-time. For in-depth explanation of this process, see Butler’s (2005) 
Chapter II. The bottom line is that by leveraging IT, a lag time of six months can 
be decreased to a few minutes. 
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b. Flow of Data 

For our system, an adaptation of the Butler (2005) system, once 
data is entered into the database, it is analyzed by the IT software and results 
are provided to the user in both numerical and graphical form. Figure 11 shows 
the flow of data, from collection by aviators, to analyses at NFS, to near-real 
reporting of results. Aviators were able to input their data directly into the online 
data collection form, an example of which is given in Figure 12 for Osan AB. 
Data analysis worksheets were developed to manipulate the aviator feedback 
received via the data collection forms. Most of the analyses were automated and 
completed via the online system. Further explanations of the data flows and 
analyses can be found in Butler’s (2005) Chapter II. 
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III. RESULTS 


A. OVERVIEW 

One of the three goals of this study was to adapt an existing web-based 
system from Butler (2005) for data collection and analysis for Air Force use at the 
CWT level. The adapted system we created can be found at 
http://wx.met.nps.navy.mil/metrics . The second goal was to implement and test 
the system, and analyze the results. The third goal was to use results to provide 
advice to AFW, ACC, and PACAF on how to alter their procedures and identify 
improvement opportunities. This chapter focuses on the third goal by discussing 
the data we collected and the results of our data analyses. The most pertinent 
results are presented in this report. Additional results fount at 

http://wx.met.nps.navv.mil/metrics/airforce/accu.html . Since the primary focus of 
this research was on the process and not the collection and analysis of an 
abundance of data, only preliminary conclusions about forecast performance and 
the operational impacts of forecasts can be determined. However, our results do 
allow us to make a thorough assessment of our main goal, the development, 
testing, and operational application of an online prototype metrics system. 

After data collection began, initial answers implied aviators misunderstood 
some of the questions in the form. For example, some completed data collection 
forms indicated aircrews were providing feedback on more than one mission per 
form, and it wasn’t clear which answers pertained to each mission, rendering the 
data unusable. Therefore, revisions were made to the wording of some 
questions. Collection resumed and data quality and quantity increased. Overall, 
approximately nine percent of the data collection forms were suspect and 
consequently left out of analyses. 

Data of sufficient quality to be used in our data analyses was collected for 
a total of 107 missions during the intensive data collection period of 9-13 January 
2006. As noted in chapter II, this figure of 107 missions is probably an 
underestimate of the actual number of aircraft that flew, because in some cases 
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multiple flights that were part of the same mission were reported on one data 
collection form. Of the 107 missions, 40 percent were flown out of the two 
participating bases in PACAF. Figure 13 is a summary of data collected, 
categorized by air base (rows) and air frame (columns). The summary describes 
the percent of total missions performed by each base, the percentage of 
squadron missions completing data collection forms, and to which major 
command (ACC or PACAF) each base was assigned. The CWT at Creech AFB 
was never able to collect data with all of the organizational transitions and other 
challenges they faced. 

Planning weather forecasts (PWFs) were defined, for this study, as 
forecasts issued prior to six hours in advance of takeoff time, while forecasts 
within six hours of takeoff were referred to as mission execution forecasts 
(MEFs). When aviators were asked the earliest time prior to mission takeoff they 
first looked at the PWF, if available, the overwhelming response was 12-24 hours 
prior. The MEF was reviewed between one and three hours prior to mission 
takeoff. 

We found that terms used in the data collection form may indicate multiple 
definitions at differing AF bases. For example, the PACAF data indicated that 
the term “mission successful” had several meanings. We concluded that, in 
general, if aviators accomplished something useful, but not necessarily what they 
intended to accomplish, the mission was deemed successful by the aviators. 
Results indicated 82 percent of all missions analyzed were deemed successful 
by aviators, while only seven percent were considered unsuccessful, and for 11 
percent of the missions success was not assessed by the aviators. 

B. PERFORMANCE AND IMPACTS RESULTS 

1. Forecast Accuracy Results 

The FAC was calculated for all weather phenomena that actually occurred 
during the missions and led aviators to infer negative mission impacts. Figure 14 
shows the FAC for forecasts of phenomena (listed on the horizontal axis) from 
which aviators inferred negative mission impacts. PWF FAC is shown in blue, 

MEF FAC in purple, and the average of PWF and MEF FAC in yellow. It is 
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important to note large values of FAC for a given phenomenon are generally 
associated with a very small sample. The FAC for turbulence, and contrails was 
zero because these weather phenomena were never inferred from the forecasts 
or observed. The FAC for in-flight icing and icing run times was zero because 
the aviators reported that the never accurately inferred these weather 
phenomena from the forecasts but did observe them. This may mean the 
forecasts were inaccurate, that the aviators did not accurately assess the 
forecast, or that they did not accurately infer the negative impacts of the 
forecasted phenomena. It is also possible that the forecast was accurate but that 
mission plan changes were made that led to the unexpected negative impacts. 
These could include small changes in mission timing and/or location, weapons, 
tactics, etc.). 

The PWF FAC shows 100 percent accuracy in forecasting visibility aloft. 
This may be misleading because visibility aloft actually occurred, and had a 
negative impact on the mission as deemed by aviators, in only four missions. 
The mean PWF FAC for cloud ceiling, cloud layers, surface visibility, and visibility 
aloft is 69.2 percent with a standard deviation of 16.9 percent between negatively 
impacting weather phenomena. The highest MEF FAC was 80 percent for 
visibility aloft. The mean MEF FAC for cloud ceiling, cloud layers, surface 
visibility, and visibility aloft was 69.2 percent with a standard deviation of 8.9 
percent between negatively impacting weather phenomena. In general, the PWF 
FAC and the MEF FAC were similar. 

The mean FAC for both PFW and MEF was 69.2 percent, which is 
comparable to the mean No Go MEF forecast accuracy in previous studies (Hinz 
2004, Jarry 2005). For example, Jarry (2005) found the mean No Go MEF FAC 
for AMC data during fiscal year 2004 was 65.6 percent. This suggests aviators 
are assessing the forecast accuracy roughly the same as weather personnel who 
use instrumentation for verification. 
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2. Probability of Detection for Negatively Impacting Weather 
Phenomena 

The POD was calculated for all weather phenomena that actually occurred 
during the mission and led aviators to infer negative mission impacts. Figure 15 
shows the POD for forecasts of phenomena (listed on the horizontal axis) from 
which aviators inferred negative mission impacts. The blue depicts PWF POD 
while the purple depicts MEF POD. It is important to note large values of POD 
for a given phenomenon are generally associated with a very small sample size. 
The POD for turbulence, and contrails was zero because these weather 
phenomena were never inferred from the forecasts or observed. The POD for in¬ 
flight icing and icing run times was zero because the aviators reported that they 
never accurately inferred these weather phenomena from the forecasts but did 
observe them. This may mean the forecasts were inaccurate, that the aviators 
did not accurately assess the forecast, or that they did not accurately infer the 
negative impacts of the forecasted phenomena. It is also possible that the 
forecast was accurate but that mission plan changes were made that led to the 
unexpected negative impacts. These could include small changes in mission 
timing and/or location, weapons, tactics, etc.). 

Overall, the PWF and MEF both had the highest POD for surface visibility 
at 86.7 percent. The mean POD for PWFs was 65.5 percent with a standard 
deviation of 15.9 percent where the PWFPOD for surface visibility was 86.7 
percent while the PWFPOD for cloud layers was 50 percent. The mean POD for 
MEFs was 65.5 percent with a standard deviation of 15.1 percent, which is 
comparable to the mean No Go MEFPOD in previous studies. The MEFPOD for 
surface visibility was 86.7 percent while the MEFPOD for cloud ceilings was 53.3 
percent. Jarry (2005) found the mean No Go MEFPOD for the AMC fiscal year 
2004 to be 67.2 percent. 

3. False Alarm Rate for Negatively Impacting Weather 
Phenomena 

The FAR was calculated for all weather phenomena that actually occurred 
during the mission and led aviators to infer negative mission impacts. Figure 16 

shows the FAR for forecasts of phenomena (listed on the horizontal axis) from 
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which aviators inferred negative mission impacts. The PWFFAR is shown in blue 
and the MEFFAR in purple. The FAR for turbulence, and contrails was zero 
because these weather phenomena were never inferred from the forecasts or 
observed. The FAC for in-flight icing and icing run times was 100 because the 
aviators reported that the never accurately inferred these weather phenomena 
from the forecasts but did observe them. This may mean the forecasts were 
inaccurate, that the aviators did not accurately assess the forecast, or that they 
did not accurately infer the negative impacts of the forecasted phenomena. It is 
also possible that the forecast was accurate but that mission plan changes were 
made that led to the unexpected negative impacts. These could include small 
changes in mission timing and/or location, weapons, tactics, etc.). 

The mean PWFFAR was 31.2 percent with a standard deviation of 36.7 
percent. The mean MEFFAR was 52.1 percent with a standard deviation of 37.7 
percent. These values are similar to previous studies as Jarry (2005) found the 
mean No Go MEFFAR for the AMC fiscal year 2004 MEF verification data was 
34.4 percent with a standard deviation between 13 CWTs of 33.3 percent. As 
previously mentioned, metrics from the two studies cannot be directly compared 
since our data was aviator-assessed and previous studies were assessed by 
weather personnel using meteorological equipment. It is, although, interesting to 
note the comparisons. 

4. Negative Impacts Inferred and Mission Pian Changes 

In 36 percent of the total missions, PWFs were inferred by aviators to 
have negative mission impacts, but in only 21 percent of the missions did 
aviators change mission plans due to the PWFs. In 39 percent of the total 
missions, MEFs were inferred by aviators to have negative mission impacts, but 
in only 21 percent of the missions did aviators change mission plans after 
interpreting the MEF. Even though aviators only changed mission plans in 
response to forecasts in 21 percent of the missions, results indicate 36 percent of 
all missions experienced weather phenomena with negative mission impacts, as 
deemed by aviators. This raises the question why aviators did not change their 
mission plans more often. Perhaps changing mission plans was not an option or 
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perhaps the aviators to take their chances and hope for weather that was 
sufficiently benign to allow them to complete their mission or alternate mission. 
This might have been the case if the aviators did not have confidence in, and 
therefore ignored, the forecasts, or if they had a back-up plan based on the 
negative mission impacts they inferred. Despite the reported negative impacts, 
the aviators assessed 82 percent of all their missions as successful. 

5. Observed Phenomena with Negative Impacts That Were/Were 
Not Forecasted 

Figure 17 shows the percent of missions that experienced phenomena (on 
the horizontal axis) with negative impacts that were/were not forecasted in the 
PWF and/or the MEF. Results indicate that in six percent of all missions, the 
PWF was interpreted to contain negatively impacting weather phenomena when 
the MEF did not, and for six percent of all missions the MEF forecasted 
negatively impacting weather when the PWF did not. Overall, the PWF and MEF 
appear to be equally accurate, although the PWF did better at cloud ceilings 
while the MEF was better at cloud layers. When turbulence occurred and 
negatively impacted the mission, it was not contained in either the PWF or MEF. 
This may imply either the forecasts were inaccurate or the aviator did not 
interpret the forecast accurately. Of the three weather phenomena having the 
largest number of negative impacts to missions, the forecasts were most 
accurate for surface visibility, although surface visibility also had the highest FAR 
(see Figure 16). Additionally, of the top three weather phenomena having 
negative impacts to missions, the forecasts did the poorest job at forecasting 
cloud layers, which indicates this is an area deserving of further attention (e.g., 
further research, education and training, etc.). 

6. Percentage of Missions for Which Negative Mission Impacts 
Occurred and Were/Were Not Indicated by the MEF 

Figure 18 shows the percent of all missions that experienced negative 
impacts (on the horizontal axis) that were/were not indicated by the MEF. Low 
level training and instrumentation training had the lowest chance of being 
indicated or forecasted by the MEF which may mean that the aviators did not 
interpret these negative mission impacts correctly or the MEFs did not accurately 
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depict the weather phenomena. It may also mean the weather was too good to 
accomplish instrumentation training because overcast conditions were necessary 
and the schedule changed after the aviator looked at the MEF and the weather 
ended up improving. Therefore, the aviators could no longer accomplish 
instrumentation training, not because of forecast or interpretation inaccuracies, 
but because the schedule changed out of their control. Additional information on 
why the mission was not completed would have helped as there is not enough 
data to make a determination. 

In 80 percent of the missions where the aviators could not see the target, 
they had been pre-warned by the MEF. Of the missions that could not takeoff as 
scheduled, the MEF was 65 percent accurate at identifying this negative impact. 
When aviators were not able to perform air-to-air training, forecasts indicated this 
impact every time. 

7. Negative Mission Impacts Inferred From Planning Weather 
Forecast and MEF 

Figure 19 shows the percent of missions for which mission plan changes 
(on the horizontal axis) were made by aviators in response to negatively 
impacting weather phenomena indicated by only the PWF (light blue), only the 
MEF (dark blue), or both the PWF and the MEF (green). Although these results 
may include multiple plan changes in response to either the MEF or PWF, the 
most significant finding here is that 26 percent of mission plan changes were 
made by aviators in response solely to the PWF while only 20 percent were 
changed solely in response to the MEF. Aviators made aircrew changes in 12 
percent of the missions, weapon changes in 17 percent of the missions, schedule 
changes in 17 percent of the missions, and mission plan changes in 11 percent 
of the missions. Another item to note is that mission plan changes (i.e. low-level 
versus high-level, strike target on the ground versus air-to-air training, etc.) were 
only made in response to the PWF. This clearly indicates that CWTs need to 
place more emphasis on PWFs. 
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8. Negative Impacts Inferred From PWF 

Figure 20 shows the number of missions for which indicated negative 
impacts (colored symbols) were inferred by aviators from PWFs of indicated 
weather phenomena (horizontal axis). Note that the forecasted phenomena 
associated with the largest number of inferred negative mission impacts were 
cloud ceilings (associated with inability to see target), cloud layers (associated 
with inability to see target), and surface visibility (associated with inability to 
takeoff or land). 

Aviators inferred from 12 PWFs that they would not be able to takeoff or 
land due to surface visibility negatively impacting their mission. From an AFW 
perspective, it is valuable to see from Figure 14 that the PWF FAC for surface 
visibility was nearly 70 percent. Aviators assessed 22 PWFs as having negative 
impacts due to surface visibility and 12 PWFs as having negative impacts due to 
cloud layers and cloud ceilings, although this could be misleading if aviators 
inferred multiple negative impacts due to the same weather phenomena. 

9. Mission Pian Changes Made Due to PWF 

Figure 21 depicts the number of missions for which indicated mission plan 
changes (colored symbols) were made based on PWFs of indicated weather 
phenomena (horizontal axis). Note that the mission schedule was changed for 
four missions in response to the PWF of surface visibility. If aviators made 
mission plan changes every time they inferred a negative mission impact would 
occur. Figure 21 would show at least the same number of mission plan changes 
for each weather phenomena as Figure 20 shows for negative impacts. This 
may be a bit confusing as multiple negative mission impacts can be inferred from 
one weather phenomena, and multiple mission plan changes can be made from 
that same phenomena. Surface visibility, for instance, was deemed to have 
negative mission impacts on, at the very least, 12 missions, but aviators did not 
make any mission plan changes for 10 of those missions. Note that in planning 
two of the missions, aviators made mission plan changes even though no 
weather phenomena were inferred to have negative mission impacts. This 
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suggests aviators did not believe the forecasts, or chose to give higher priority to 
factors other than the forecasts in planning their missions. 

10. Negative Impacts Inferred From MEF 

Figure 22 shows the number of missions for which indicated negative 
impacts (colored symbols) were inferred by aviators from MEFs of indicated 
weather phenomena (horizontal axis). Note that the inability to takeoff or land 
was inferred for 14 missions based on forecasts of surface visibility in the MEFs 
for those missions. Surface visibility once again was assessed to have the most 
negative mission impacts on takeoffs and landings, and negative impacts from 
this phenomenon were inferred in 14 MEFs. Negative mission impacts were 
inferred for a total of 23 missions due to surface visibility in the MEF, and cloud 
layers and clouds ceilings had the next highest inference of negative mission 
impacts at 14 each. The MEFFAC for surface visibility is just over 60 percent. 
The following section will discuss whether or not aviators made mission plan 
changes in response to the forecasts. It is important to note that aviators do not 
always have the ability to make mission plan changes in response to the 
forecasts. 

11. Mission Pian Changes Made Due to MEF 

Figure 23 depicts the number of missions for which indicated mission plan 
changes (colored symbols) were made based on MEFs of indicated weather 
phenomena (horizontal axis). As an example from this figure: weapon changes 
were made for four missions based on the cloud layer forecasts in the MEFs for 
those missions. If aviators made mission plan changes every time they inferred 
a negative mission impact would occur. Figure 23 would show at least the same 
number of mission plan changes for each weather phenomena as Figure 22 
shows for each negative impact. We focus on surface visibility once again, as it 
was deemed to have negative mission impacts on, at the very least, 14 missions 
(Figure 22), but aviators did not make any mission plan changes for 10 of those 
missions (Figure 23), suggesting they did not believe the forecast for at least four 
missions. Results also indicate cloud ceilings and cloud layers were inferred to 
have negative mission impacts on at least 11 missions each (Figure 22) and no 
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mission plan changes were made during five of those 11 missions (Figure 23). 
This suggests cloud ceiling and cloud layer forecasts were believed less than 
surface visibility forecasts. 

12. Weather Phenomena Resulting in Unsuccessful Mission 

After seeing how aviators interpret forecasts and then make mission plan 
changes, it is important to highlight which weather phenomena caused the small 
percentage of unsuccessful missions. Figure 24 shows the percent of missions 
deemed unsuccessful due to negative impacts from weather phenomena 
(horizontal axis). Seven percent of all missions analyzed were deemed 
unsuccessful by aviators. Five percent of all mission failures were due to surface 
visibility, and two percent were due to visibility aloft. Results from Figures 20 and 
21 suggest aviators did not believe surface visibility forecasts during four 
missions. If aviators had made mission plan changes each time the forecast 
indicated negative mission impacts, the outcome might have been different. 
Results also show the MEF FAR for surface visibility was nearly 40 percent, 
which may support the reason aviators did not make mission plan changes. 

13. Positive Mission Contribution to Successfui Missions 

Figure 25 depicts the percent of missions for which a positive mission 
contribution (PMC) was made by the PWF (seen in blue) or MEF (seen in 
purple). PMC criteria are summarized in the blue text box within the figure and 
are described in detail in Chapter II. Percentages are based on the number of 
missions deemed successful by aviators. For all bases, 10.2 percent of 
successful missions received a PMC from their PWFs, and 10.2 percent of 
successful missions received a PMC from their MEFs. The PMC for Beale AFB, 
Nellis AFB, and Whiteman AFB was zero because there was little negatively 
impacting weather phenomena identified in the forecasts by aviators, thus the 
opportunity to have a PMC was extremely low. The greatest PMC to mission 
success was Osan AB, where the PWF and MEF each had roughly a six percent 
PMC to all successful missions. As mentioned above, Osan AB also flew 40 
percent of all missions analyzed. 
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14. Potential Positive Mission Contribution to All Missions 

Figure 26 shows the percent of all missions for which a potential positive 
mission contribution (PPMC) was made by the PWF (seen in blue) or the MEF 
(seen in purple). PPMC criteria are summarized in the blue text box within the 
figure and are described in detail in Chapter II. For all bases, 4.7 percent of all 
missions received a PPMC from their PWFs, and 3.7 percent of missions 
received a PPMC from their MEFs. The PPMC for Beale AFB, Nellis AFB, and 
Whiteman AFB was zero because there was little negatively impacting weather 
phenomena identified in the forecasts by aviators, thus the opportunity to have a 
PPMC was extremely low. The Kunsan AB PPMC was zero because aviators 
typically made changes when they perceived negative mission impacts would 
occur due to weather. Again, results indicate Osan AB had the highest PPMC at 
3.7 percent. This suggests aviators had the potential to achieve their primary 
mission in 3.7 percent of all missions had they made mission plan changes when 
they perceived negative mission impacts would occur due to MEFs. 

Overall, PWFs had a higher PPMC of 4.7 percent, when compared to the 
MEFs at 3.7 percent, while the PMC was the same for PWFs and MEFs (see 
figure 25). This indicates the importance of PWFs to mission planning and 
execution. 

C. NPS AVIATOR SURVEY RESULTS 

Of the 36 NPS pilots and navigators surveyed, 15 provided lengthy 
feedback to our survey (Figure 7). These survey results confirmed the 
identification from our ACC and PACAF data collection of surface visibility, cloud 
ceilings, and cloud layers as the three weather phenomena resulting in the most 
negative mission impacts, and therefore the three of most interest to them in 
forecasts. The survey results also suggest most aviators only look at forecasts 
within 24 hours of takeoff and usually only within six hours, unless the mission is 
“other than normal” training such as a “check ride” where the aviators are 
upgrading their flying qualifications. Most aviators have a secondary mission 
plan when stepping to their aircraft to fly and also have the flexibility to execute 
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the secondary plan while in flight. Thus, they will deem many missions 
“successful” even though the primary objective was not accomplished. 

After data analyses were completed, four NPS aviators participated in a 
round-table discussion concerning the results. All aviators that had worked with 
weather personnel embedded in the flying squadron felt the support was far 
superior to their experiences with non-embedded weather personnel. Aviators 
felt embedded weather personnel had a better feel for the environment within 
which they operate and also had a better handle on mission-impacting weather. 
Aviators also felt the weather support provided by embedded CWT personnel 
was better tailored to their needs. 

When asked about our findings which indicate PWFs might be under 
utilized, NPS aviators did not have consistent feedback as to whether or not they 
remembered using or even seeing a CWT-issued PWF. Feedback indicated 
aviators used various sources for planning weather purposes, such as television 
and internet sources. 

Those aviators who had spent at least two years as instructors in Air 
Education and Training Command (AETC) felt they spent considerably more time 
planning a mission in AETC than they did when flying in ACC or PACAF units. 
Aviators indicated AETC missions are driven by a strict syllabus and when 
planning a mission, if there was a high probability of not achieving the directives 
in the syllabus, they would cancel instead of flying anyway and trying to 
accomplish something else. This confirms our assessment that the definition of a 
“successful” mission varies widely among flying units. It also suggests that a 
future study of the performance and operational impacts of forecasts issued by 
AETC CWTs would be useful. 

D. SUMMARY 

The general consensus from the ACC and PACAF CWTs was that the 
data collection form was too long and therefore aircrews were reluctant to 
complete it. The form was also not structured in a flow consistent with aircrew 
thought processes after they had just completed a mission. Several data 
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collection forms were found where aircrews started to answer mission planning 
questions, the first questions on the form, with mission execution answers and 
then when they reached the mission execution questions, realized what they had 
done, and used arrows and side notes to explain. A proposed re-structured data 
collection form is discussed in Chapter IV. 

Since the focus of this thesis was to develop a prototype, proof of concept 
system, in some cases there was not enough data to adequately assess results. 
However, we did attempt to calculate all of the performance and impacts metrics 
described in chapter II with the data we collected. However, for six of the 
impacts metrics, we felt the data quality and/or quantity was not sufficient to 
justify calculating them. 

Of the 107 missions analyzed, 40 percent were from PACAF flying units 
and 60 percent from ACC units. In 82 percent of the missions, aviators deemed 
the mission successful, which we interpreted to mean that if the aircrew was not 
able to accomplish the primary mission, then an alternate plan was successfully 
executed. In the seven percent of unsuccessful missions due to weather 
phenomena, surface visibility was the primary cause. The primary weather 
phenomena that caused negative impacts to missions were surface visibility, 
cloud layers, and cloud ceilings. Of these three, the forecasts did the poorest job 
at predicting, and/or aviators did the poorest job at interpreting, cloud layers. 

Forecast performance metrics indicated the mean FACs for the PWFs and 
the MEFs were 69.2 percent as assessed by aviators, which is slightly higher 
than the mean No Go MEF FAC in previous studies where weather personnel 
assessed the accuracy (e.g., Jarry 2005). Results indicated a mean POD of 65.5 
percent for both the PWF and MEF, which is slightly lower than previous studies; 
for example, a mean No Go MEF POD of 67.2 percent from Jarry (2005). The 
mean PWF FAR was 31.2 percent and the mean MEF FAR was 52.1 percent. 
Jarry (2005) found a mean No Go MEF FAR at 34.4 percent for AMC in fiscal 
year 2004. 
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Operational impacts metrics indicated both the PWF and the MEF had a 
PMC of 10.2 percent. This may not be so important in a training environment, 
but in a war-time environment where the aviators must accomplish their primary 
mission, this would most likely be important. Additionally, PWFs had a PPMC to 
all missions analyzed of 4.7 percent while the MEF PPMC was 3.7 percent. This 
suggests more emphasis should be placed on PWFs, especially when several of 
the NPS aviators indicated they often times looked for planning weather from 
sources other than their CWTs because they were not aware their CWTs issued 
PWFs. 

Many of the values we obtained in our calculation of forecast performance 
and operational impacts metrics were similar to those obtained in prior studies 
using much larger data sets (e.g., Hinz 2004, Jarry 2005). This suggests that, 
although we collected a limited amount of data, and collected it from aviators, not 
CWT personnel, the data quantity and quality were still sufficient to allow us 
calculate useful metrics. 

The online data collection, analysis and reporting part of our system was 
not sufficiently tested by the aviators to assess its effectiveness from the aviator’s 
perspective. However, we suspect that web-based collection by aviators may be 
more of a challenge than completing paper copies because of the need to have a 
computer available to every aviator at the time of their debriefs. Collecting data 
in locations where weather personnel are not fully integrated into the flying 
squadrons was a challenge due to the interface needed, and NPS aviators 
verified the strain on CWT-aviator coordination when weather personnel are not 
embedded in the flying squadrons. However, CWT manning plays a huge role in 
whether or not full integration is accomplished and we saw a reflection of low 
CWT manning in the lower amounts of data collected. 

There were sharp contrasts in the amount of data collected from ACC and 
PACAF units, primarily due to differences in the occurrence of negatively 
impacting weather, which was not as common for the ACC units during our data 
collection period. 
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IV. SUMMARY, DISCUSSIONS, AND RECOMMENDATIONS 


A. SUMMARY OF PROCEDURES AND RESULTS 

Many new and existing challenges face AFW personnel in today’s 
resource-limited and changing environment. Accurate weather forecasts are 
imperative in aviation planning and execution of combat operations. To ensure 
forecast accuracy and operational relevancy, quantitative analyses of forecast 
performance and operational impacts are essential, but should be done in the 
most efficient and effective manner possible. For this reason, an existing web- 
based, near-real time system for collecting and analyzing weather and aviation 
data was adapted for use by AFW, ACC, and PACAF. Consistent with Figure 9, 
the data came from planning and execution forecasts as well as observed 
weather phenomena (left side of Figure 9), and from aviation planning and 
execution feedback (right side of Figure 9),. Although the data was limited, we 
were able to calculate metrics which appear to provide useful information about 
overall forecast performance and operational impacts. Due to lack of negatively 
impacting weather phenomena at the ACC bases, much of the data on negatively 
impacting weather came from PACAF bases. However, our process and results 
can be applied to all AFW CWTs. 

1. Goals Accomplished 

The first goal we set out to accomplish was to adapt a system already in 
use by the U.S. Navy for collecting and analyzing data in near-real time. This 
was much more challenging than anticipated because our data collection and 
analyses were done by aviators and not weather personnel, which resulted in the 
rewrite of much of the computer code. 

Our second goal was to implement and test the system with seven ACC 
and PACAF CWTs. Six of the seven CWTs were able to participate in data 
collection. But, due to IT challenges and time constraints, the online portion of 
the system was not operational until much of the data had already been 
collected. However, we did receive feedback from two CWTs indicating it would 
not have been a good idea to have aviators input the data directly into an online 
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system instead of filling out a paper copy of the data collection form. CWT 
personnel directly entering data into the online form is a better option. Of the six 
CWTs that provided data, three felt they gained valuable feedback on ways they 
could improve aviator support through the process of coordinating the data 
collection form with the aviators. 

Our third goal was to use the results from system implementation to 
provide AFW, ACC, and PACAF operations advice on how to alter procedures 
and allocation of resources to better accomplish goals. We ended up calculating 
five quantitative metrics of forecast performance and 15 quantitative metrics of 
operational impacts. 

2. Results 

A total of 107 missions were analyzed; 82 percent deemed successful, 
seven percent deem unsuccessful, and 11 percent not assessed by aviators. 
Aviators inferred negative mission impacts from 36 percent of the PWFs, but only 
made mission plan changes to 21 percent of the missions in response to the 
PWFs. Aviators inferred negative mission impacts from 39 percent of the MEFs, 
but again, only made mission plan changes to 21 percent of the missions in 
response to these MEFs. However, aviators encountered negatively impacting 
weather phenomena in 36 percent of their missions. 

Forecast performance metrics indicated a mean FAC of 69.2 percent for 
both PWFs and MEFs. The mean POD for both PWFs and MEFs was 65.5 
percent. Results, for PWFs and MEFs, deviated when the FAR was calculated. 
The mean FAR for PWFs was 31.2 percent while the mean MEF FAR was 52.1 
percent. This gives PWFs a slight edge over MEFs, which is important to note 
since CWTs spend significantly more time formulating MEFs than PWFs. 

Operational impacts were measured by relating MEF performance metrics 
to operational performance metrics. We determined that surface visibility, cloud 
ceilings, and cloud layers were the weather phenomena that most adversely 
affected missions. Surface visibility was cited as the cause in 71 percent of the 
missions deemed unsuccessful due to weather, which represents five percent of 
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all missions. Aircrews made four types of mission plan changes, but only 
changed the mission type (e.g. low-level, refueling, and air-to-air) due to PWFs, 
never MEFs. This indicates PWFs may need to be given more time and attention 
by CWTs and aircrews. 

PWFs and MEFs both provided a PMC to 10.2 percent of all successful 
missions. As explained in Chapter II, a forecast has a PMC when the aviator 
deems negative mission impacts will result from weather phenomena, makes 
mission plan changes based on the forecast, and subsequently achieves their 
primary mission. Also explained in Chapter II is a PPMC. A PPMC occurs when 
an accurate forecast implies negative mission impacts, aviators make no mission 
plan changes, and then the primary mission is not accomplished. The PWFs had 
a PPMC in 4.7 percent of all missions, while the MEFs had a PPMC of 3.7 
percent. Once again, this suggests more emphasis needs to be placed on 
PWFs. 

The NPS aviator survey validated findings of the top three weather 
phenomena needing further research and training. The high percentage of 
successful missions was justified by NPS aviators who explained that missions 
they had flown while in ACC and PACAF were deemed successful if they 
accomplished something useful, even if they did not accomplish their primary 
mission. They reported that it was common for aircrews to have a secondary 
mission plan and the flexibility to execute, if needed. However, aviators who had 
flown in both AETC and either ACC or PACAF described a notable difference 
when flying in AETC, versus ACC or PACAF. At ETC, mission plans are closely 
tied to a syllabus, with no secondary plan, so success is determined by following 
the syllabus. NPS aviators also provided valuable feedback regarding the CWT 
products. Most of the NPS aviators were not confident they knew how, when, 
and where to access all of the CWT products, especially PWFs. Some were 
unaware that the CWTs at the flying bases they had been assigned issued formal 
PWFs, others received planning weather by phoning the CWT, and yet others 
looked up planning weather via other sources on the web. 


55 



3. Meeting AFW Challenges and Vision 

Implementing an online system such as the one developed and tested in 
this research project, will help improve mission efficiency and effectiveness of 
CWTs. The results of this project will help enable CWTs to compensate for not 
having sufficient personnel to fully integrate themselves into all the flying 
squadrons they support. 

Step 11 of the MEFP as outlined in chapter four of AFMAN 15-129 directs 
the MEF be a product for which CWTs receive continuous feedback from 
aviators. The system in this research project provides continuous aviator 
feedback. The first task in step 12 of the MEFP as outlined in chapter four of 
AFMAN 15-129 calls for the implementation of systematic procedures to analyze 
and measure accuracy/relevancy of products provided to customers. Step two 
directs operator debriefs, preferably face-to-face, but when this is not possible it 
directs CWTs to employ other feedback methods. Step five directs CWTs to 
conduct operations verification on established “Go/No Go” thresholds and step 
six requires the development and application of metrics for process improvement 
(AFMAN 15-129 2004). Our system meets or could readily be adapted to meet, 
all of these directives. 

Guidance in AFI 15-114 deems operational verification of the MEF as the 
“single most important mission-oriented, operational effectiveness assessment 
requirement for CWTs.” It also calls for a system that “automates all metrics, 
from data collection, aggregation, to data quality control. The end-state will be 
an automated web-based system with the capability to provide ad hoc analyses 
and reports (assessments) for all levels of AFW support” (AFI 15-114). Our 
system meets the AFI guidance above. 

The AFAA report released September 2005 found that Air Force 
personnel did not consistently and completely obtain customer feedback and 
identify improvement opportunities, and recommended that AFW establish 
standardized procedures for obtaining mission operator feedback on weather 
forecasts (AFAA 2005). Our system is a solution to this finding. 
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The AFWSPV set the course for transformation starting with fiscal year 
2008. The process we are proposing in this thesis takes steps, however small, 
toward three of the eight vectors. It creates automated decision tools which can 
further be exploited (vector 6), it helps determine where weather information is 
integrated into decision cycles at the base level and below (vector 7), and 
although not directly, it enables similar feedback results that could be 
accomplished by embedding weather personnel into flying squadrons (vector 8) 
(AFWSPV 2004). 

B. RECOMMENDATIONS 

The task list for CWTs is lengthy and manpower challenges make 
achieving many of the tasks unrealistic. Implementation of the prototype system 
we have adapted for usage at Air Force bases will enable AFW to alleviate some 
of the challenges facing CWTs. Specifically, providing this prototype as an 
appendix to AFI 15-114 will provide CWTs an example of an automated system 
that is already working. It also eliminates CWTs from having to train personnel 
on metrics calculations. 

Although missions vary drastically from fighter aircraft to bombers, to 
reconnaissance and surveillance, etc., generic data collection forms can be 
drafted for each platform. Our results indicate our data collection form was too 
long and the order in which questions were asked needed to be changed. 
Questions pertaining to the actual mission need to be asked first because 
aviators complete the forms immediately after they have flown their mission. 
Figure 27 is a revised generic data collection form for fighter aircraft. We 
recommend similar forms be created for differing platforms. Once these data 
collection forms are produced, they can be used in our prototype system for data 
collection. 

When we started coordination with the original seven CWTs, it became 
apparent there is a need in the AFW community to formalize training on getting to 
know your customer. We recommend that a formal training process and 
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documents be developed and implemented, and that CWT personnel be given 
enough overlap with those they replace to allow for ample orientation to the flying 
squadrons they will be supporting. 

The NPS aviators could not praise embedded weather support in the flying 
squadrons enough. Aviators indicated the value added when weather personnel 
were available in-person at the flying squadron was immeasurable. Further, they 
felt that once a weather person experienced the environment in which aviators 
operate, they were able to understand weather requirements better and 
established credibility with the aviators. We recommend AFW leadership 
reinforce flying squadron integration and CWT manning. Feedback from CWTs 
during our coordination and data collection processes indicated that air-to-ground 
platforms are the most “weather needy” and emphasis should be placed on 
manning CWTs accordingly. Further, we recommend an operational 
effectiveness study be conducted to compare the quantitative impact of having 
embedded weather personnel in the flying squadrons to not having embedded 
weather personnel. 

Several of our results suggest that aviators are not familiar with many of 
the terms AFW personnel use. The feedback we received also suggests AFW 
personnel need to educate aviators better on the types of products available and 
when and where aviators can find these products. 

Results indicate AFW forecasts have the ability to provide positive mission 
impacts when aviators follow forecasts and make mission plan changes 
accordingly. It appears PWFs can make a difference in whether or not aviators 
achieve their primary mission. So we recommend that aviators better incorporate 
weather products earlier in their planning process, at least 12-24 hours prior to 
takeoff. 

C. FUTURE WORK 

Work already in progress at the NPS is adapting this web-based, near-real 
time system for collecting and analyzing METOC and aviator data from U.S. 
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Navy aircraft carriers. The same system being adapted across U.S. military 
service lines will improve joint service cooperation. 

Feedback from CWTs and NFS aviators stressed the large number of 
different agencies wanting feedback from aviators after mission execution. Thus, 
it would be useful to incorporate weather data collection forms into other existing 
pilot debrief systems. Perhaps airfield flight and weather could combine their 
data collection efforts into one database. Semi-automated feedback needs to 
continue, but will not be successful unless it is tied into an automated system that 
is already established. Evidently, such a system is already setup at Whiteman 
AFB called Patriot Excalibur. Unfortunately, the system is not accessible from a 
non-“.mil” website therefore limiting usage. 

Adapting our prototype system to combat operations, such as Operation 
Iraqi Freedom, would provide valuable “war-time” information. This would need 
to be accomplished over a secure network which would involve a lot more time 
due to limited availability to communicate with the data collectors in theater, but 
would be extremely valuable. 

The biggest challenge we faced was getting data. The fast-paced aviator 
environment proved to be a challenging one to tap into. Unless aviators can be 
provided qualitative and quantitative justification as to why it is advantageous for 
them to provide feedback, it will continue to be the most challenging part of 
operational verification for the AFW community. 
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APPENDIX 


FIGURES 


Schematic Timelines Linking 
Aviation Operations and Weather Support 



Figure 1. Schematic timeline linking aviation operations to weather support. 

Timeline begins on left with long range planning that occurs three months 
prior to mission takeoff to the actual mission execution on the right. 
Linkage provides framework for determining data collection opportunities. 
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Operations Process Data to be Collected 



Figure 2. Flow chart for planning, executing, and debriefing air operations (blue 
boxes), and corresponding data collection needed for analyses of the 
performance and operational impacts of weather forecasts (yellow boxes). 
The operations flow chart indicates the types of weather products aviators 
use at different stages in their planning and execution. 
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Figure 3. Flow chart depicting all avenues mission planning and execution can take 
from receipt of a weather forecast to completion of a successful mission. 
Flow chart traces back the forecast-related steps that led to a successful 
mission. Steps are those that involve the planning weather forecast 
(PWF, bottom row), mission changes made in response to PWF, second 
row up), mission execution forecast (MEF, third row up), mission changes 
made in response to MEF (fourth row up), and actual weather experienced 
during the mission (fifth row up). 
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Figure 4. Flow chart depicting all avenues mission planning and execution can take 
from receipt of a weather forecast to completion of an unsuccessful 
mission. Flow chart traces back the forecast-related steps that led to an 
unsuccessful mission. Steps are those that involve the planning weather 
forecast (PWF, bottom row), mission changes made in response to PWF, 
second row up), mission execution forecast (MEF, third row up), mission 
changes made in response to MEF (fourth row up), and actual weather 
experienced during the mission (fifth row up). 
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Generic Data Collection Form 


Mission Identification 


Mission #/Call Sign: 

MO/DATTR_ Take-off Time (Zulu). 

PLANNING 


1. How many hours prior to your flight did you first check the Combat Weather 
Team (CWT) weather? 

48, 36, 24,12,6,0 

2. How many hours prior to your flight would you have checked the CWT weather 
if it were available? 

72, 60,48, 24,12,0 

3. What negative mission impacts were forecasted in the planning weather ? 
Won’t see target 

Won’t be able to lock-on target 

Won’t be able to do air-to-air training 

Won’t be able to take-off or land 

Won’t be able to refuel 

Won’t be able to do instrumentation training 

Won’t be able to do touch and gos 

Won’t be able to do low-level training 

None 

4. What weather phenomena led to these forecasted negative impacts? 

Cloud Ceiling 

Cloud Layers 

Thunderstorms 

Rain 

Visibility at Surface 

Visibility Aloft 

Turbulence 

Icing 

X-wind 

None 

5. What changes in mission plans were made due to planning weather 
forecasting negative impacts? 

Aircrew 

Weapon 

Schedule (i.e. time of mission) 
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EXECUTION 


6. How many hours prior to your flight did you check the Mission Execution 
Forecast (MEF)? 

6. 5, 4, 3,2, 1,0 

7. What negative mission impacts were forecasted in MEF? 

Won’t see target 

Won’t be able to lock-on target 

Won’t be able to do air-to-air training 

Won’t be able to take-off or land 

Won’t be able to refuel 

Won’t be able to do instrumentation training 

Won’t be able to do touch and gos 

Won’t be able to do low-level training 

None 

8. What weather phenomena led to these MEF negative impacts? 

Cloud Ceiling 

Cloud Layers 

Thunderstorms 

Rain 

Visibility at Surface 

Visibility Aloft 

Turbulence 

Icing 

X-wind 

None 

9. What additional changes in mission plans were made due to MEF negative 
impacts? 

Aircrew 

Weapon 

Schedule (i.e. time of mission) 

Type (i.e. high alt, instrumentation, strike, low-level) 

None 

10. What negative impacts due to weather actually occurred during mission? 
Couldn’t take-off 

Couldn’t see target 
Couldn’t refuel 

Couldn’t perform low-level training 
Couldn’t perform air-to-air training 
Couldn’t perform instrumentation flight 
None 
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11. What weather phenomena led to these negative impacts during your 
mission ? 

Cloud Ceiling 
Cloud Layers 
Thunderstorms 
Rain 

Visibility at Surface 

Visibility Aloft 

Turbulence 

Icing 

X-wind 

None 

12. How are you using Target Acquisition Weapons Software (TAWS)? 

Not using (skip to question #21) 

Mission planning 

Execution 

Both 

13. Does TAWS have your weapon sensor modeled? 

Yes 

No 

14. What are you using TAWS for? 

IR 

TV 

Laser 

NVG predictions 

15. Did you change your weapons load or tactics based on TAWS predictions? 
Yes 

No 

16. How were the TAWS predictions? 

Pessimistic 

Optimistic 

Accurate 

17. Did you download weather from AFWA or get a TAWS file from your CWT? 
AFWA 

CWT 

18. Was the TAWS weather accurate? 

Yes 

No 
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19. Would you use TAWS again? 

Yes, but it could be improved by_ (insert your suggestions) 

No 


20. Rate the prediction software in the areas below using the following scale: 
1-not useful, 2-somewhat useful but not required, 3-useful, 4-mission essential 


Pre-mission: 

a. Route selection 12 3 

b. Target viewing azimuth 12 3 

c. Target area tactics 12 3 

Inflight: 

a. Low level navigation 12 3 

b. Target area tactics 12 3 

c. Target identification 12 3 

d. Threat avoidance 12 3 

e. Sensor cross-check techniques 12 3 


4 

4 

4 


4 

4 

4 

4 

4 


21. What did you intend to accomplish on your mission? 

Low-level training 

High-altitude training 

Instrumentation training 

Air-to-Air Combat training 

Air-to-Air Refueling 

Strike Target on Ground 


22. What did you actually accomplish on your mission? 

Low-level training 

High-altitude training 

Instrumentation training 

Air-to-Air Combat training 

Air-to-Air Refueling 

Strike Target on Ground 

Nothing 


23. 


a. 


b. 


c. 


What was your overall impression of the actual mission? 


□ 

Mission successful 

□ 

Mission unsuccessful 

□ 

Changed mission due to 

□ 

No change in mission due to 

□ 

planning weather 

□ 

planning weather 

Changed mission due to MEF 

No mission change due to MEF 


d. 


□ 

□ 

□ 


No negative weather impacts forecasted 
Negative weather impacts forecasted accurately 
Negative weather impacts forecasted inaccurately 


e. 



No WX impact to mission 


□ 
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Negative WX impact to mission 






Figure 5. Generic data collection form used to gain feedback on planning, execution 
and TAWS data from aviators after mission execution. 


IRTSS Questionnaire 

Nellis AFB is the lead ACC unit for software that creates Sensor 
Prediction Products (SPP). Infrared Target Scene Simulation (IRTSS) is 
the latest and greatest SPP tool. Thanks for your time and for helping 
make the program better. 

Which sensor[s] did you employ on your mission? 

Was the IRTSS prediction helpful in your mission planning? 

Was the IRTSS prediction a realistic representation of what you saw on 
your cockpit display? 

Did IRTSS represent 'feeder features' (roads, rivers, etc) into the target 
area correctly? 

Did IRTSS represent the target area correctly? 

Did the IRTSS prediction influence you to adjust/alter tactical decisions 
before mission execution? 

What is one enhancement that if made to IRTSS would make the product 
more useful to you? 

Figure 6. Infrared target scene simulation (IRTSS) data collection form used to gain 
feedback from aviators after mission execution. 
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Naval Postgraduate School Aviation Survey Form 

1. Please provide a brief history of your fiying experience to inciude 
airframes fiown and number of hours in that airframe. 

2. in generai, how many days prior to a mission did you first iook at the 
weather? How did you receive this weather? (internet, base weather 
station, weather channei) What were you iooking for? (generai sky 
coverage, thunderstorms, precipitation, fog, soiar/iunar data) 

3. in generai, how fiexibie were you at making changes to the mission 
pian? Piease eiaborate on how maintenance and perhaps airfieid 
operations piay into this. Discuss weapons, targets, scheduiing 
ranges, refueiing, scheduiing iow-ievei routes, different category 
piiots. 

4. in your opinion, what are the most important parts of the MEF? 
(Mission Execution Forecast issues by the Combat Weather Team at 
your instaiiation) Which weather phenomena (icing, turbuience, 
visibiiity at surface, etc.) in your opinion, caused the most mission 
pianning changes on your part? 

5. From your fiying experiences, what does Air Force Weather (AFW) do 
weii and what couid AFW stand to put time and effort into? 

6. in generai, how much faith have you had in the weather forecasts 
you’ve received? Did you find the quaiity of the weather support you 
received changed based on weather fiight ieadership? Person to 
person? 

7. When you first get to a new iocation, do you tend to pay more 
attention to the weather forecast versus having been at a iocation for 
a year where you’ve experienced aii four seasons and in the back of 
your mind can picture what you’ii see based on a sateiiite photo? 
Seems iike common sense, but just want to confirm. 

8. How weii is weather integrated into operations? Long range ciimate 
forecasts? Pianning weather? MEF? 

Figure 7. Naval Postgraduate School aviator survey form. Information from this 
survey was used to validate and interpret results of ACC and PACAF data 
analyses. 
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Ave # Msns/day 

Ave length of Msn 
Mission Type 

A-10 

Thunderboit ii 

B-1B 

Lancer 

B-2 

Spirit 

F-16 

Fighting Faicon 

U-2 

UAV (Unmanned 
Aeriai Vehicie) 

Beaie AFB, CA 





5 Ave Msns/day 

5 Hrs/Msn 

S&R 


Creech AFB, NV 






9 Ave Msns/day 

8 Hrs/Msn 

S&R 

Dyess AFB, TX 


10 Ave Msns/day 

6 Hrs/Msn 

B 





Kunsan AB, ROK 




18 Ave Msns/day 
1.8 Hrs/Msn 

AA, AG 



Nellis AFB, NV 

8 Ave Msns/day 

2 Hrs/Msn 

AA, AG, CAS 



8 Ave Msns/day 

2 Hrs/Msn 

AA, AG 



Osan AB, ROK 

10 Ave Msns/day 

3 Hrs/Msn 

AA, AG, CAS 



15 Ave Msns/day 
1.5 Hrs/Msn 

AA, AG 



Whiteman AFB, MO 



9 Msns/day 

4 Hrs/Msn 

B 





S&R=Surveillance and Reconnaissance B=Bombing 

AA=Air-to-Air CAS=Close Air Support 

AG=Air-to-G round 


Figure 8. Summary of missions flown during data collection timeframe, categorized 
by air base (rows) and air frame (columns). Information on missions 
describes; average number of missions flown per day, average length of 
mission (hours), and mission type (see key at bottom of table). 
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Figure 9. Schematic data and methods flow chart showing the data collected (top 
row), performance analysis (middle row), and assessment of operational 
impacts (bottom row). 
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Data Analyses Calculations 

(Based on Kunsan, Nellis, and Osan data collection form) 

A. Number of total missions, successful missions and unsuccessful missions 
analyzed 

1. Count the number of total missions entered into the database, the 
number of missions deemed successful in #23a, and the number of 
missions deemed unsuccessful in #23a 

2. Display results in numeric form showing total missions, successful 
missions, and unsuccessful missions 

B. Percentage of missions where aviator would have checked the planning 
weather earlier had it been available. 

1. If #2 is greater than #1, then divide the number of times this exists 
by the total number of missions analyzed and multiply by 100 

2. Display results in a bar graph with time (in hours) increments (72, 
60, 48, 24, 12, 0) on the x-axis and frequency of occurrence on the y- 
axis 

C. Percentage of missions deemed unsuccessful and the aviators would 
have checked the planning weather earlier had it been available. 

1. If #23a shows mission unsuccessful and #2 is greater than #1, then 
divide the number of times this exists by the total number of missions 
analyzed and multiply by 100 

2. Display results in a pie chart with the total number unsuccessful 
missions as the total pie and the missions where aviators would have 
checked the planning weather earlier had it been available as a piece 
of the pie 

D. Percentage of missions considered successful even though aviators did 
not actually accomplish what they intended to. 

1. If #23a shows mission successful and #21 and #22 do not have the 
same answers, then divide the number of times this exists by the total 
number of missions analyzed and multiply by 100 

2. Display results as a pie chart with total number of successful 
missions as the total pie and the missions where aviators didn’t 
accomplish what they intended to as a piece of the pie 
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E. Percentage of missions that were successful even though negative 
weather impacts were forecasted in MEF, but aviators did not make any 
mission changes due to the MEF. 

1. If #23a shows mission successful and #7 shows anything but 
“None” and either #9 shows “None” or #23c shows “No mission change 
due to MEF”, then divide the number of times this exists by the total 
number of successful missions and multiply by 100 

2. Display results as a pie chart with the total number of successful 
missions being the total pie and the piece of the pie representing 
missions where negative weather impacts were forecasted in the MEF, 
but no mission plans were changed 

F. Correlation between weather phenomena and negative mission impacts 

1. Match responses to #3 and #7 to the corresponding responses 
other than “None” for #4 and #8. 

2. Display results as a chart with the mission impacts on one axis, 
weather phenomena on the other axis, and frequency of occurrence in 
the corresponding square 

G. Percentage of missions the aircrew checked the MEF 6, 5, 4, 3, 2, 1, 0 
hours prior to takeoff 

1. Count each response to #6 and divide it by the total number of 
missions 

2. Display results as a bar graph with the hours on the x-axis and 
percent of missions on the y-axis 

H. Percentage of missions successful where there were changes in mission 
plans due to planning weather and aviators actually accomplished what they 
intended to (Positive Mission Contribution due to PWF) 

1. If #23a shows “mission successful” and #5 shows anything other 
than “None” and #22 has at least the same options as #21 and #23b 
shows “Changed mission due to planning weather”, divide the number 
of times this exists by the total number of missions analyzed and 
multiply by 100 

2. Display results as a pie chart with letter I, where the whole pie is the 
total number of successful missions analyzed and a piece of the pie is 
the PMC due to PWF 
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I. Percentage of missions successful where there were changes in mission 
plans due to MEF and aviators actually accomplished what they intended to 
(Positive Mission Contribution due to MEF) 

1. If #23a shows “mission successful” and #9 shows anything other 
than “None” and #22 has at least the same options as #21 and #23c 
shows “Changed mission due to MEF”, divide the number of times this 
exists by the total number of missions analyzed and multiply by 100 

2. Display results as pie chart with letter H, where the whole pie is the 
total number of successful missions analyzed and a piece of the pie is 
the PMC due to MEF 

J. Correlation between weather phenomena forecasted and resulting mission 
plan changes 

1. Most frequent responses other than “None” to #4 and #8 and the 
corresponding responses to #5 and #9 

2. Display results as a chart with the mission plan changes on one 
axis, weather phenomena on the other axis, and frequency of 
occurrence in the corresponding square 

K. Weather phenomena in MEF resulting in the most unsuccessful missions 

1. Identify the correlation between a response of “Mission 
unsuccessful” in #23a and the corresponding answer to #8 other than 
“None” 

2. Display each weather phenomena on x-axis and # of unsuccessful 
missions on y-axis 

L. Percentage of missions a negative impact due to weather actually 
occurred when MEF did/didn’t forecast it 

1. If #10 shows anything other than “None” then see if #7 shows a 
corresponding response and if it does, then it “was forecasted”: if it 
does not, it “was not forecasted.” Divide both the “was forecasted” and 
the “was not forecasted” responses for each impact by the total 
occurring and multiply by 100 to get percentages 

2. Display each negative mission impact on x-axis and the percentage 
of missions in which it corresponded to the #7 response on the y-axis 

M. Percentage of missions a particular mission plan was changed due to 
planning weather and then changed again due to MEF 

1. Responses in #5 other than “None” that correspond to the same 
response in #9 

2. Display mission plan changes on x-axis and on y-axis show three 
separate percentages: # changed due to planning weather, # changed 
due to MEF, # plan was changed both due to planning weather and 
then again due to MEF 
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N. Percentage of missions where mission was unsuccessful and operator felt 
the MEF was inaccurate. 

1. If #23a shows “Mission unsuccessful” and #8 responses do not 
match #11 responses and #23d shows either “No negative weather 
impacts forecasted” or “Negative weather impacts forecasted 
inaccurately”. Divide the number of times this exists by the number of 
unsuccessful missions and multiply by 100 

2. Display results numerically 

O. The most common weather phenomena actually occurring that resulted in 
mission being successful/unsuccessful. 

1. #11 responses other than “None” that correspond to #23a “mission 
successful/mission unsuccessful” 

2. Display the weather phenomena on the x-axis and number of 
missions on the y-axis. Display successful missions in one color and 
unsuccessful missions in another color on the bar graph 

P. Percentage of each weather phenomena negatively impacting the 
mission, but was/wasn’t forecasted in MEF 

1. If response to #11 is anything other than “None” and it wasn’t in #8 
responses 

2. Display the weather phenomena on the x-axis and number of MEFs 
on the y-axis. Display both when they correspond and when they do 
not in different colors on the bar graph 

Q. Percentage of missions a weather phenomena actually occurred, but 
wasn’t in the MEF(planning weather) even though it was in planning 
weather(MEF) or was not in either or was in both 

1. If response to #11 is anything other than “None” and it wasn’t in #8 
(#4) responses, but it was in #4 (#8) or it wasn’t in either #8 or #4 or it 
was in both #8 and #4. 

2. Display the weather phenomena on the x-axis and number of MEFs 
(planning weather forecasts) on the y-axis. Display each of the four 
situations in different colors. 

Target Acquisition Weapon Software (TAWS) 

R. Percentage of missions that used TAWS 

1. Count the number of responses for #13 that responded “Not using” 
and subtract if from the total number of forms submitted and then 
divide this number by the total number of missions and multiply by 100. 

2. Display results in numeric form. 
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S. How is TAWS being used? 

1. Responses for #13 other than “Not using” and divide each response 
by the totai number of TAWS users and muitipiy by 100. 

2. Dispiay resuits in a bar graph with “Mission planning”, “Execution”, 
and “Both” on the x-axis and percentage of TAWS usage on the y-axis 

T. Percentage of time TAWS had the weapon sensor modeled (didn’t have 
the weapon sensor modeled) and the TAWS predictions were accurate, 
pessimistic, or optimistic? 

1. Responses to #17 that matched a “Yes”(“No”) response to #14 

2. Display results as a bar graph with the TAWS predictions 
(pessimistic, optimistic, accurate) on the x-axis and the number of 
times these responses corresponded to the weapon sensor being 
modeled (“Yes”) in one color and the number of times these responses 
corresponded to the weapon sensor not being modeled (“No”) in 
another color 

U. What is TAWS being used for? 

1. Count the number of times each response for #15 is marked and 
divide it by the total number of TAWS users. Multiply this by 100 to get 
a percentage. 

2. Display each of the responses to #15 on the x-axis and the 
percentage of time they are using TAWS for it on the y-axis 

V. Percentage of missions TAWS weather was accurate/inaccurate when it 
came from the CWT versus AFWA 

1. Count the number of times each response for #18 is marked and 
the percentage of time each response corresponded to a #19 response 
of “Yes” or “No” 

2. Display the responses to #18 on the x-axis and the percentage of 
time they were “Yes” to #19 in one color and the percentage of time 
they were “No” in #19 in another color.. .the percentages are on the y- 
axis 

W. Percentage of TAWS users that would use it again when their weapon 
sensor was/wasn’t modeled 

1. Count the number of times each response for #20 is marked and 
the percentage of time this corresponded to either answer to #14. 

2. Display each of the responses to #20 on the x-axis and the 
percentage of time their weapon sensor was modeled in one color and 
the percentage of time their weapon sensor was not modeled in 
another color 
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X. Percentage of time the weapons load or tactics were changed based on 
TAWS predictions 

1. Count the number of times each response for #16 is marked and 
divide it by the total number of TAWS users...multiply this by 100 to 
get a percentage. 

2. Display each of the responses to #16 on the x-axis and the 
percentage of time they occurred on the y-axis 

Y. Usefulness of TAWS 

1. Count the number of times each response for #21 is marked and 
divide it by the total number of TAWS users...multiply this by 100 to 
get a percentage. 

2. Display each of the responses to #21 on the x-axis and the 
corresponding percentage of usefulness (1-4) on the y-axis in four 
different colors...may need to do two bar graphs, one for “Pre-mission” 
and one for “In-flight” 

Figure 10. Description of calculations performed to analyze ACC and PACAF data, 
plus instructions for displaying the results of the analyses. Based on data 
collected from Kunsan AB, Nellis AFB, and Osan AB. 
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to AFW and operations 
leadership 


ACC a PACAF CWTs 


Figure 11. Flow of data, from collection by aviators, to analyses at Naval 
Postgraduate School, to results reported to leadership. (Adapted from 
Butler 2005) 
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2. How many hours prior to your flight would you have checked the C\VT ^^'eather if it ^^-ere available? 

<"72 <"60 <"48 <"24 <"12 <" 6 <" 0 

3. \Vhat negative mission impacts were forecasted in the planning weather ? 

Select all that ^pty 

r Won't see target 
n Won't be able to lock-on target 
r Won't be able to do air-to-air traaung 
r Won't be able to take-off or land 
I” Won't be able to refuel 
r Won't be able to do mstrumentanon training 
r Won't be able to do touch and go's 


Figure 12. Online data collection form for Osan AB available online at 
http://wx.met.nps.navv.mil/metrics/index.html . Data collected is put into a 
Naval Postgraduate School database where it is then analyzed, and from 
which metrics reports are issued. 
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% of Total Missions 
% of Sqdn Msns 
MAJCOM 

A-10 

Thunderbolt II 

B-1B 

Lancer 

B-2 

Spirit 

F-16 

Fighting Falcon 

U-2 

UAV (Unmanned 
Aerial Vehicle) 

Beale AFB, CA 





5% 

30% 

ACC 


Creech AFB, NV 






0% 

Dyess AFB, TX 


17% 

60% 

ACC 





Kunsan AB, ROK 




14% 

35% 

PACAF 



Nellis AFB, NV 

8% 

10% 

ACC 



0% 



Osan AB, ROK 

16% 

50% 

PACAF 



24% 

50% 

PACAF 



Whiteman AFB, MO 



16% 

40% 

ACC 





Figure 13. Summary of data collected during data collection timeframe, categorized 
by air base (rows) and air frame (columns). Information on missions 
describes: percent of total missions, percent of squadron missions 
completing data collection forms, and major command (Air Combat 
Command or Pacific Air Forces). 
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120.00%t 



CloudCeiling CloudLayers SUrfaceVis VisAloft In-fligWIcing IcIngRunTinrES Turbulence Contrails 

Forecasted tniiathcr Phenomena 


Figure 14. Forecast accuracy (FAC) for forecasts of phenomena (listed on horizontal 
axis) from which aviators inferred negative mission impacts. Planning 
weather forecast FAC (blue), mission execution forecast FAC (purple), 
average of PWF and MEF FAC (yellow). Note: large values of FAC for a 
given phenomenon are generally associated with a very small sample 
size. 
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100 , 00 % 



Forecasted weather Phenomena 


Figure 15. Probability of detection (POD) for forecasts of phenomena (listed on 
horizontal axis) from which aviators inferred negative mission impacts. 
Planning weather forecast POD (blue), mission execution forecast POD 
(purple). Note: large values of POD for a given phenomenon are 
generally associated with a very small sample size. 
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120 . 00 % 



Times 

Forecasted Weather Phenomena 


Figure 16. False alarm rate (FAR) for forecasts of phenomena (listed on horizontal 
axis) from which aviators inferred negative mission impacts. Planning 
weather forecast FAR (blue), mission execution forecast FAR (purple). 
Note: large values of FAR for a given phenomenon are generally 
associated with a very small sample size. 
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16 


CO 

c 

o 

CO 

CO 


I Not forecasted in PWF or MEF (Total: 17%) 

I Forecasted in PWF but not in MEF (Total: 6%) 

I Not forecasted in PWF but was in MEF (Total: 6%) 
I Forecasted in both PWF and MEF (Total: 30%) 



ceiling layers t-storms rain sfc vis vis aloft turb icing x-wind 


Figure 17. Percent of missions that experienced phenomena (horizontal axis) with 
negative impacts that were/were not forecasted in the planning weather 
forecast (PWF) and/or the mission execution forecast (MEF). 


m Was Indicated by 
MEF 


H Was Not Indicated by 
" MEF 



Can't See Target Can't Takeoff Can't Refuel NoLLTrng NoA2ATrng NolnstTmg 


Figure 18. Percent of missions that experienced negative impacts (horizontal axis) 
that were/were not indicated by the mission execution forecast (MEF). 
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Percentage of Missions 


Figure 19. 


12 


Based only on PWF 
Based only on MEF 
Based on PWF and MEF 



Percent of missions for which mission plan changes (horizontal axis) were 
made by aircrews in response to negatively impacting weather 
phenomena indicated by planning weather forecast only (PWF, light blue), 
mission execution forecast only (MEF, dark blue), or both PWF and MEF 
(green). 
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Figure 20. Number of missions for which indicated negative impacts (colored 
symbols) were inferred by aviators from planning weather forecasts of 
indicated weather phenomena (horizontal axis). Note that the forecasted 
phenomena associated with the largest number of inferred negative 
mission impacts were cloud ceiling (associated with inability to see target), 
cloud layers (associated with inability to see target), and surface visibility 
(associated with inability to takeoff or land). 
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Figure 21. Number of missions for which indicated mission plan changes (colored 
symbols) were made based on planning weather forecasts of indicated 
weather phenomena (horizontal axis). Example: the mission schedule 
was changed for four missions based on the PWF forecast of surface 
visibility. 
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Figure 22. Number of missions for which indicated negative impacts (colored 
symbols) were inferred by aviators from mission execution forecasts of 
indicated weather phenomena (horizontal axis). Example: inability to 
takeoff or land was inferred for 14 missions based on forecasts of surface 
visibility in the MEFs for those missions. 
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Figure 23. Number of missions for which indicated mission plan changes (colored 
symbols) were made based on mission execution forecasts of indicated 
weather phenomena (horizontal axis). Example: weapons changes were 
made for four missions based on the cloud layer forecasts in the MEFs for 
those missions. 
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Percentage of Missions 


8 



Figure 24. Percent of missions deemed unsuccessful due to negative impacts from 
indicated weather phenomena (horizontal axis). Seven percent of all 
missions analyzed were deemed unsuccessful by aviators. 
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Percentage of Successful Missions 


8 . 00 % 


700% 


□ PWF PMC 
I MEF PMC 


6 . 00 % 


PWF PMC = 10.2% 
MEF PMC = 10.2% 


500% 


PMC: forecast implies negative mission 
impact, mission plan changes, and 
primary mission accomplished 


4.00% 


300% 


2 . 00 % 


1 00% 


0 . 00 % 




Beale AFB Dyess AFB Kunsan AB Nellis AFB Osan AB 


\Miiteman AFB 


Figure 25. Percent of missions for which a positive mission contribution (PMC) was 
made by the planning weather forecast (PWF, blue) or mission execution 
forecast (MEF, purple). PMC criteria are summarized in blue text box 
within figure and are described in detail in Chapter II. Percentages based 
in part on number of missions deemed successful by aviators. For all 
bases, 10.2% of successful missions received a PMC from their PWFs, 
and 10.2% of missions received a PMC from their MEFs. 
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Percentage of All Missions 


4.00% 


□ PWF PPMC 
3.50% - □ MEF PPMC 


3.00% 


2.50% 


2 . 00 % 


1.50% 


1 . 00 % 


0.50% 


0 00% 


PWF PPMC = 4.7% 
MEF PPMC = 3.7% 


PPMC: accurate forecast, 
no mission plan changes, 
and primary mission not 
accomplished 



Beale AFB Dyess AFB Kunsan AB Nellis AFB Osan AB 


lASilteman AFB 


Figure 26. Percent of missions for which a potential positive mission contribution 
(PPMC) was made by the planning weather forecast (PWF, blue) or 
mission execution forecast (MEF, purple). PPMC criteria are summarized 
in blue text box within figure and are described in detail in Chapter II. 
Percentages based in part on number of missions deemed unsuccessful 
by aviators. For all bases, 4.7% of missions received a PPMC from their 
PWFs, and 3.7% of missions received a PPMC from their MEFs. 
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Generic Fighter Aircraft Data Coiiection Form 


Call slgn:_ 


_Date (mm/dd/yyyy):_ 


_Take-off tlme(zulu):_Land time (zulu):_ 


PRE-EXECUTION 

1. How many hours prior to your flight did you check the weather provided by the weather 
station (MEF-mission execution forecast)? 

□ 6 nS 04 nS u2 □! nO 

2. What weather phenomena in the MEF were inferred to cause negative mission impacts? 

□ Surface Visibility □ Cloud Ceiling 

□ Clouds Layers □ Visibility Aloft 

□ Other_ □ None 

3. What negative mission impacts did you infer from the MEF? 

□ Won’t be able to take-off □ Won’t be able to refuel 

□ Won’t be able to strike target □ Won’t be able to do low-level training 

□ Won’t be able to do air-to-air training □ Won’t be able to land 

□ Other_ □ None 

4. What mission plan changes due to the MEF were made prior to your flight ? 

□ Aircrew change □ Schedule (i.e. time of mission) 

□ Too late to make any changes □ Weapon 

□ Type (e.g. low-level, refueling) □ Other_ 

□ None 

EXECUTION 

5. What negative impacts due to weather actually occurred during the mission ? 

□ Couldn’t take-off □ Couldn’t refuel 

□ Couldn’t strike target □ Couldn’t do low-level training 

□ Couldn’t do air-to-air training □ Couldn’t land 

□ Other_ □ None 


6. Was the MEF accurate? 

□ Yes □ No, what was inaccurate? 

□ Surface Visibility 

□ Clouds Layers 

□ Other__ 


□ Cloud Ceiling 

□ Visibility Aloft 

□ None 


7. What did you intend to accomplish on your mission? 

□ Low-level training □ Air-to-air combat training 

□ Strike target on ground □ Air-to-air refueling 

□ Instrumentation training □ Other_ 


8. What did you actually accomplish on your mission? 


□ Low-level training 

□ Strike target on ground 

□ Instrumentation training 

□ Nothing 


□ Air-to-air combat training 

□ Air-to-air refueling 

□ Other_ 


93 



9. What was your overall impression of the mission? 

a) □ Mission successful 

□ Mission unsuccessful 

b) □ No negative impacts forecasted 

□ Negative impacts forecasted inaccurately 

□ Negative impacts forecasted accurately 

c) □ No weather impact to mission 

□ Weather negatively impacted mission 

d) □ Changed mission due to MEF 

□ No mission change due to MEF 

e) □ Changed mission due to planning weather 

□ No mission change due to planning weather 

PLANNING WEATHER 

10. How many days prior to your flight did you first check the planning weather provided 
by the base weather station? 

□ 6 nS D 4 nS a 2 nl nO 

11. Would you have checked the weather earlier if it were available? 

□ Yes □ No 

12. What mission plan changes due to planning weather were made >12hrs prior to your 
take-off? 

□ Aircrew change □ Schedule (i.e. time of mission) 

□ Weapon □ Type (e.g. low-level, refueling) 

□ Other_ □ None 


TAWS 

13. How are you using Target Acquisition Weapon Software (TAWS)? 

□ Not using (you are finished with data collection form...thanks for your feedback) 

□ Mission planning 

□ Execution 

□ Both 

14. Does TAWS have your weapon sensor modeled? 

□ Yes □ No 

15. What are you using TAWS for? 

□ IR □ TV □ Laser □ NVG predictions 

16. Did you change your weapons load or tactics based on TAWS predictions? 

□ Yes □ No 

17. How were the TAWS predictions? 

□ Pessimistic □ Optimistic □ Accurate 

18. How did you get your weather for TAWS? 

□ Download from AFWA □ File from the CWT 
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19. Was the TAWS weather accurate? 

□ Yes □ No 

20. Would you use TAWS again? 

□ Yes □ No 

21. Rate the prediction software in the areas below using the following scale: 
1-not useful, 2-somewhat useful but not required, 3-useful, 4-mission essential 

Pre-mission: 

a. Route selection 

b. Target viewing azimuth 

c. Target area tactics 

Inflight: 

a. Low level navigation 12 3 4 

b. Target area tactics 12 3 4 

c. Target identification 12 3 4 

d. Threat avoidance 12 3 4 

e. Sensor cross-check techniques 12 3 4 


Figure 27. Proposed generic fighter data collection form for future studies. 


12 3 4 
12 3 4 
12 3 4 
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